This project contains Python implementations of various climate index algorithms which provide a geographical and temporal picture of the severity of precipitation and temperature anomalies useful for climate monitoring and research. This code is currently considered ‘Beta’ as NIDIS performs additional testing and verification. If you are currently running another implementation of the algorithms, how do these Python indices compare? Please help us by reporting feedback to firstname.lastname@example.org.
The following indices are provided:
- SPI, Standardized Precipitation Index, utilizing both gamma and Pearson Type III distributions
- SPEI, Standardized Precipitation Evapotranspiration Index, utilizing both gamma and Pearson Type III distributions
- PET, Potential Evapotranspiration, utilizing either Thornthwaite or Hargreaves equations
- PDSI, Palmer Drought Severity Index
- scPDSI, Self-calibrated Palmer Drought Severity Index
- PHDI, Palmer Hydrological Drought Index
- Z-Index, Palmer moisture anomaly index (Z-index)
- PMDI, Palmer Modified Drought Index
- PNP, Percentage of Normal Precipitation
This Python implementation of the above climate index algorithms is being developed with the following goals in mind:
- to provide an open source software package to compute a suite of climate indices commonly used for climate monitoring, with well documented code that is faithful to the relevant literature and which produces scientifically verifiable results
- to provide a central, open location for participation and collaboration for researchers, developers, and users of climate indices
- to facilitate standardization and consensus on best-of-breed climate index algorithms and corresponding compliant implementations in Python
- to provide transparency into the operational code used for climate monitoring activities at NCEI/NOAA, and consequent reproducibility of published datasets computed from this package
- to incorporate modern software engineering principles and programming best practices
The installation and configuration described below is performed using a bash shell, either on Linux, Windows, or MacOS.
Download the code
Download the zip file from the links at the top of this page. Unzip and move into the source directory.
$ cd climate_indices
Within this directory, there are six subdirectories:
climate_indices: main computational package
tests: unit tests for the main package
scripts: scripts and supporting utility modules used to perform processing of indices computations on climatological datasets (typically grids or US climate division datasets in NetCDF)
example_inputs: example/reference datasets that can be used as inputs to the processing scripts
notebooks: Jupyter Notebooks describing the internals of the computational modules
docs: documentation files
- results: empty output directory for example output files
- extra_scripts: other scripts that allow extra functions including data ingest, comparisons scripts for results...etc.
Configure the Python environment
This project’s code is written in Python 3. It’s recommended to use either the Miniconda3 (minimal Anaconda) orAnaconda3 distribution. The below instructions will be Anaconda specific (although relevant to any Python virtualenv ), and assume the use of a bash shell.
A new Anaconda environment can be created using the conda environment management system that comes packaged with Anaconda. In the following examples, we’ll use an environment named indices_env (any environment name can be used instead of indices_env) which will be created and populated with all required dependencies through the use of the provided
First create the Python environment:
$ conda create -n indices_env
The environment created can now be ‘activated’:
$ conda activate indices_env
Once the environment has been activated then subsequent Python commands will run in this environment where the package dependencies for this project are present.
Now the package can be added to the environment along with all required modules (dependencies) via python and the setup.py script:
$ python setup.py install
Initially, all tests should be run for validation:
$ export NUMBA_DISABLE_JIT=1
$ python setup.py test
$ unset NUMBA_DISABLE_JIT
If you run the above from the main branch and get an error then please send a report and/or add an issue, as all tests should pass.
The Numba environment variable is set/unset in order to bypass Numba’s just-in-time compilation process, which significantly reduces testing times.
Included are scripts which interact with the core computational package to compute one or more climate indices. These are
process_grid.py which is used to compute indices from gridded NetCDF datasets, and
process_divisions.py which is used to compute indices from US climate division NetCDF datasets. Both of these scripts are located within the “scripts” directory in the zipped file at the top of this page.
These Python scripts are written to be run via bash shell commands, i.e.
$ python process_grid.py <options>
|index||Which of the available indices to compute. Valid values are ‘spi’, ‘spei’, ‘pnp’, ‘scaled’, and ‘palmers’. ‘scaled’ indicates all three scaled indices (SPI, SPEI, and PNP) and ‘palmers’ indicates all Palmer indices (PDSI, PHDI, PMDI, SCPDSI, and Z-Index).|
|periodicity||The periodicity of the input dataset files. Valid values are ‘monthly’ and ‘daily’. Note: only SPI and PNP support daily inputs.|
|netcdf_precip||Input NetCDF file containing a precipitation dataset, required for all indices except for PET. Requires the use of var_name_temp in conjunction so as to identify the NetCDF’s precipitation variable.|
|var_name_precip||Name of the precipitation variable within the input precipitation NetCDF.|
|netcdf_temp||Input NetCDF file containing a temperature dataset, required for PET. If specified in conjunction with an index specification of SPEI or Palmers then PET will be computed and written as a side effect, since these indices require PET. This option is mutually exclusive with netcdf_pet/var_name_pet, as either temperature or PET is required as an input (but not both) when computing SPEI and/or Palmers. Requires the use of var_name_temp in conjunction so as to identify the NetCDF’s temperature variable.|
|var_name_temp||Name of the temperature variable within the input temperature NetCDF.|
|netcdf_pet||Input NetCDF file containing a PET dataset, required for SPEI and Palmers. This option is mutually exclusive with netcdf_temp/var_name_temp, as either temperature or PET is required as an input (but not both) when computing SPEI and/or Palmers. Requires the use of var_name_pet in conjunction so as to identify the NetCDF’s PET variable.|
|var_name_pet||Name of the PET variable within the input PET NetCDF.|
|netcdf_awc||Input NetCDF file containing an available water capacity, required for Palmers. Requires the use of var_name_awc in conjunction so as to identify the NetCDF’s AWC variable.|
|awc_var_name||Name of the available water capacity variable within the input AWC NetCDF.|
|output_file_base||Base file name for all computed output files. Each computed index will have an output file whose name will begin with this base name plus the index’s abbreviation plus a month scale (if applicable), connected with underscores, plus the ‘.nc’ extension. For example for SPI at 3-month scale the resulting output files will be named <output_file_base>_spi_gamma_03.nc and <output_file_base>_spi_pearson_03.nc.|
|scales||Time step scales over which the PNP, SPI, and SPEI values are to be computed. Required when the index argument is ‘spi’, ‘spei’, ‘pnp’, or ‘scaled’.|
|calibration_start_year||Initial year of the calibration period.|
|calibration_end_year||Final year of the calibration period (inclusive).|
Input Data Requirements
This software enables the user to calculate several drought indices using any gridded NetCDF dataset. However, there are certain specifications for input files that vary based on input type:
|Potential Evapotranspiration (PET)
|Miscellaneous (I.e., Available Water Content or AWC)
Time-series Dimension Specifics:
When relevant, the time dimension should include full years (padded with NaN or Null values) and leap years with 366 days for daily files.
Example Command Line Invocations
Example Output files
All example output files will be in a netCDF file format following the Climate and Forecast metadata conventions. Each output file will have three dimensions, time, longitude, and latitude. The “results” directory included in the examples below serves as the output directory and will be empty until an example is properly conducted. These netCDF files can be opened and visualized with several publicly available software including NOAA’s Weather and Climate Toolkit and NASA Goddard’s Panoply. For the SPI and SPEI examples below there will be periods of no-data matching the supplied time-scales used, due to the nature of the algorithms used. For example, a 3-month SPI starting on January 1st requires 3-months of data to create a statistically valid SPI value, so the data values will not start until April 1st. The output files do not trim the initial supplied time-scale, the first 3-months in the example above, of no-data. The length of the time dimension will still match that of the input data.
$ python process_grid.py --index pet --periodicity monthly --netcdf_temp ../example_inputs/nclimgrid_lowres_tavg.nc --var_name_temp tavg --output_file_base ../results/nclimgrid_lowres --calibration_start_year 1951 --calibration_end_year 2010
The above command will compute PET (potential evapotranspiration) using the Thornthwaite method from an input temperature dataset (in this case, the reduced resolution nClimGrid temperature dataset provided as an example input). The input dataset is monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The output file will be /results/nclimgrid_lowres_pet.nc.
$ python process_grid.py --index spi --periodicity daily --netcdf_precip ../example_inputs/cmorph_lowres_daily_conus_prcp.nc --var_name_precip prcp --output_file_base ../results/cmorph_lowres_daily_conus --scales 30 90 --calibration_start_year 1998 --calibration_end_year 2016
The above command will compute SPI (standardized precipitation index, both gamma and Pearson Type III distributions) from an input precipitation dataset (in this case, the reduced resolution CMORPH precipitation dataset provided in the example inputs directory). The input dataset is daily data and the calibration period used will be Jan. 1st, 1998 through Dec. 31st, 2016. The index will be computed at 30-day and 90-day timescales. The output files will be /results/cmorph_lowres_daily_conus_spi_gamma_30.nc, /results/cmorph_lowres_daily_conus_spi_gamma_90.nc, /results/cmorph_lowres_daily_conus_spi_pearson_30.nc, and /results/cmorph_lowres_daily_conus_spi_pearson_90.nc.
$ python process_grid.py --index spi --periodicity monthly --netcdf_precip ../example_inputs/nclimgrid_lowres_prcp.nc --var_name_precip prcp --output_file_base ../results/nclimgrid_lowres --scales 6 12 --calibration_start_year 1951 --calibration_end_year 2010
The above command will compute SPI (standardized precipitation index, both gamma and Pearson Type III distributions) from an input precipitation dataset (in this case, the reduced resolution nClimGrid precipitation dataset provided in the example inputs directory). The input dataset is monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The index will be computed at 6-month and 12-month timescales. The output files will be /results/nclimgrid_lowres_spi_gamma_06.nc, /results/nclimgrid_lowres_spi_gamma_12.nc, /results/nclimgrid_lowres_spi_pearson_06.nc, and /results/nclimgrid_lowres_spi_pearson_12.nc.
$ python process_grid.py --index spei --periodicity monthly --netcdf_precip ../example_inputs/nclimgrid_lowres_prcp.nc --var_name_precip prcp --netcdf_pet ../example_inputs/nclimgrid_lowres_pet.nc --var_name_pet pet --output_file_base ../results/nclimgrid_lowres --scales 9 18 --calibration_start_year 1951 --calibration_end_year 2010
The above command will compute SPEI (standardized precipitation evapotranspiration index, both gamma and Pearson Type III distributions) from input precipitation and potential evapotranspiration datasets (in this case, the reduced resolution nClimGrid precipitation and PET datasets provided in the example inputs directory). The input datasets are monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The index datasets will be computed at 9-month and 18-month timescales. The output files will be /results/nclimgrid_lowres_spi_gamma_09.nc, /results/nclimgrid_lowres_spi_gamma_18.nc, /results/nclimgrid_lowres_spi_pearson_09.nc, and /results/nclimgrid_lowres_spi_pearson_18.nc.
$ python process_grid.py --index palmers --periodicity monthly --netcdf_precip ../example_inputs/nclimgrid_lowres_prcp.nc --var_name_precip prcp --netcdf_pet ../example_inputs/nclimgrid_lowres_pet.nc --var_name_pet pet --netcdf_awc ../example_inputs/nclimgrid_lowres_soil.nc --var_name_awc awc --output_file_base ../results/nclimgrid_lowres --calibration_start_year 1951 --calibration_end_year 2010
The above command will compute the Palmer drought indices: PDSI (original Palmer Drought Severity Index), PHDI (Palmer Hydrological Drought Index), PMDI (Palmer Modified Drought Index), Z-Index (Palmer Z-Index), and SCPDSI (Self-calibrated Palmer Drought Severity Index) from input precipitation, potential evapotranspiration, and available water capacity datasets (in this case, the reduced resolution nClimGrid precipitation, PET, and AWC datasets provided in the example inputs directory). The input datasets are monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The output files will be /results/nclimgrid_lowres_pdsi.nc, /results/nclimgrid_lowres_phdi.nc, /results/nclimgrid_lowres_pmdi.nc, /results/nclimgrid_lowres_scpdsi.nc, and /results/nclimgrid_lowres_zindex.nc.