You are here

Climate and Drought Indices in Python

Download Software

Climate indices for everyone

This project contains Python implementations of various climate index algorithms which provide a geographical and temporal picture of the severity of precipitation and temperature anomalies useful for climate monitoring and research.  This code is currently considered ‘Beta’ as NIDIS performs additional testing and verification.  If you are currently running another implementation of the algorithms, how do these Python indices compare?  Please help us by reporting feedback to nidis.questions@noaa.gov.

The following indices are provided:

  • SPI, Standardized Precipitation Index, utilizing both gamma and Pearson Type III distributions
  • SPEI, Standardized Precipitation Evapotranspiration Index, utilizing both gamma and Pearson Type III distributions
  • PET, Potential Evapotranspiration, utilizing either Thornthwaite or Hargreaves equations
  • PDSI, Palmer Drought Severity Index
  • scPDSI, Self-calibrated Palmer Drought Severity Index
  • PHDI, Palmer Hydrological Drought Index
  • Z-Index, Palmer moisture anomaly index (Z-index)
  • PMDI, Palmer Modified Drought Index
  • PNP, Percentage of Normal Precipitation

This Python implementation of the above climate index algorithms is being developed with the following goals in mind:

  • to provide an open source software package to compute a suite of climate indices commonly used for climate monitoring, with well documented code that is faithful to the relevant literature and which produces scientifically verifiable results
  • to provide a central, open location for participation and collaboration for researchers, developers, and users of climate indices
  • to facilitate standardization and consensus on best-of-breed climate index algorithms and corresponding compliant implementations in Python
  • to provide transparency into the operational code used for climate monitoring activities at NCEI/NOAA, and consequent reproducibility of published datasets computed from this package
  • to incorporate modern software engineering principles and programming best practices

Getting started

The installation and configuration described below is performed using a bash shell, either on Linux, Windows, or MacOS.

Windows users will need to install and configure a bash shell in order to follow the usage shown below. We recommended either babun or Cygwin for this purpose.

Download the code

Download the zip file from the links at the top of this page.  Unzip and move into the source directory.

$ cd climate_indices

Within this directory, there are six subdirectories:

  • climate_indices: main computational package
  • tests: unit tests for the main package
  • scripts: scripts and supporting utility modules used to perform processing of indices computations on climatological datasets (typically grids or US climate division datasets in NetCDF)
  • example_inputs: example/reference datasets that can be used as inputs to the processing scripts
  • notebooks: Jupyter Notebooks describing the internals of the computational modules
  • docs: documentation files
  • results: empty output directory for example output files
  • extra_scripts: other scripts that allow extra functions including data ingest, comparisons scripts for results...etc.

Configure the Python environment

This project’s code is written in Python 3. It’s recommended to use either the Miniconda3 (minimal Anaconda) orAnaconda3 distribution. The below instructions will be Anaconda specific (although relevant to any Python virtualenv ), and assume the use of a bash shell.

A new Anaconda environment can be created using the conda environment management system that comes packaged with Anaconda. In the following examples, we’ll use an environment named indices_env (any environment name can be used instead of indices_env) which will be created and populated with all required dependencies through the use of the provided setup.py file.

First create the Python environment:

$ conda create -n indices_env

The environment created can now be ‘activated’:

$ source activate indices_env

Once the environment has been activated then subsequent Python commands will run in this environment where the package dependencies for this project are present.

Now the package can be added to the environment along with all required modules (dependencies) via python and the setup.py script:

$ python setup.py install

Testing

Initially, all tests should be run for validation:

$ export NUMBA_DISABLE_JIT=1

$ python setup.py test

$ unset NUMBA_DISABLE_JIT

If you run the above from the main branch and get an error then please send a report and/or add an issue, as all tests should pass.

The Numba environment variable is set/unset in order to bypass Numba’s just-in-time compilation process, which significantly reduces testing times.

Indices Processing

Included are scripts which interact with the core computational package to compute one or more climate indices. These are process_grid.py which is used to compute indices from gridded NetCDF datasets, and process_divisions.py which is used to compute indices from US climate division NetCDF datasets. Both of these scripts are located within the “scripts” directory in the zipped file at the top of this page.

These Python scripts are written to be run via bash shell commands, i.e.

$ python process_grid.py <options>



Option Description
index Which of the available indices to compute. Valid values are ‘spi’, ‘spei’, ‘pnp’, ‘scaled’, and ‘palmers’. ‘scaled’ indicates all three scaled indices (SPI, SPEI, and PNP) and ‘palmers’ indicates all Palmer indices (PDSI, PHDI, PMDI, SCPDSI, and Z-Index).
periodicity The periodicity of the input dataset files. Valid values are ‘monthly’ and ‘daily’. Note: only SPI and PNP support daily inputs.
netcdf_precip Input NetCDF file containing a precipitation dataset, required for all indices except for PET. Requires the use of var_name_temp in conjunction so as to identify the NetCDF’s precipitation variable.
var_name_precip Name of the precipitation variable within the input precipitation NetCDF.
netcdf_temp Input NetCDF file containing a temperature dataset, required for PET. If specified in conjunction with an index specification of SPEI or Palmers then PET will be computed and written as a side effect, since these indices require PET. This option is mutually exclusive with netcdf_pet/var_name_pet, as either temperature or PET is required as an input (but not both) when computing SPEI and/or Palmers. Requires the use of var_name_temp in conjunction so as to identify the NetCDF’s temperature variable.
var_name_temp Name of the temperature variable within the input temperature NetCDF.
netcdf_pet Input NetCDF file containing a PET dataset, required for SPEI and Palmers. This option is mutually exclusive with netcdf_temp/var_name_temp, as either temperature or PET is required as an input (but not both) when computing SPEI and/or Palmers. Requires the use of var_name_pet in conjunction so as to identify the NetCDF’s PET variable.
var_name_pet Name of the PET variable within the input PET NetCDF.
netcdf_awc Input NetCDF file containing an available water capacity, required for Palmers. Requires the use of var_name_awc in conjunction so as to identify the NetCDF’s AWC variable.
awc_var_name Name of the available water capacity variable within the input AWC NetCDF.
output_file_base Base file name for all computed output files. Each computed index will have an output file whose name will begin with this base name plus the index’s abbreviation plus a month scale (if applicable), connected with underscores, plus the ‘.nc’ extension. For example for SPI at 3-month scale the resulting output files will be named <output_file_base>_spi_gamma_03.nc and <output_file_base>_spi_pearson_03.nc.
scales Time step scales over which the PNP, SPI, and SPEI values are to be computed. Required when the index argument is ‘spi’, ‘spei’, ‘pnp’, or ‘scaled’.
calibration_start_year Initial year of the calibration period.
calibration_end_year Final year of the calibration period (inclusive).

Example Command Line Invocations

 

Example Output files

All example output files will be in a netCDF file format following the Climate and Forecast metadata conventions. Each output file will have three dimensions, time, longitude, and latitude. The “results” directory included in the examples below serves as the output directory and will be empty until an example is properly conducted. These netCDF files can be opened and visualized with several publicly available software including NOAA’s Weather and Climate Toolkit and NASA Goddard’s Panoply. For the SPI and SPEI examples below there will be periods of no-data matching the supplied time-scales used, due to the nature of the algorithms used. For example, a 3-month SPI starting on January 1st requires 3-months of data to create a statistically valid SPI value, so the data values will not start until April 1st. The output files do not trim the initial supplied time-scale, the first 3-months in the example above, of no-data. The length of the time dimension will still match that of the input data.

PET monthly

$ python process_grid.py --index pet --periodicity monthly --netcdf_temp ../example_inputs/nclimgrid_lowres_tavg.nc --var_name_temp tavg --output_file_base ../results/nclimgrid_lowres --calibration_start_year 1951 --calibration_end_year 2010

The above command will compute PET (potential evapotranspiration) using the Thornthwaite method from an input temperature dataset (in this case, the reduced resolution nClimGrid temperature dataset provided as an example input). The input dataset is monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The output file will be /results/nclimgrid_lowres_pet.nc.

SPI daily

$ python process_grid.py --index spi  --periodicity daily --netcdf_precip ../example_inputs/cmorph_lowres_daily_conus_prcp.nc --var_name_precip prcp --output_file_base ../results/cmorph_lowres_daily_conus --scales 30 90 --calibration_start_year 1998 --calibration_end_year 2016

The above command will compute SPI (standardized precipitation index, both gamma and Pearson Type III distributions) from an input precipitation dataset (in this case, the reduced resolution CMORPH precipitation dataset provided in the example inputs directory). The input dataset is daily data and the calibration period used will be Jan. 1st, 1998 through Dec. 31st, 2016. The index will be computed at 30-day and 90-day timescales. The output files will be /results/cmorph_lowres_daily_conus_spi_gamma_30.nc, /results/cmorph_lowres_daily_conus_spi_gamma_90.nc/results/cmorph_lowres_daily_conus_spi_pearson_30.nc, and /results/cmorph_lowres_daily_conus_spi_pearson_90.nc.

SPI monthly

$ python process_grid.py --index spi --periodicity monthly --netcdf_precip ../example_inputs/nclimgrid_lowres_prcp.nc --var_name_precip  prcp --output_file_base ../results/nclimgrid_lowres --scales 6 12 --calibration_start_year 1951 --calibration_end_year 2010

The above command will compute SPI (standardized precipitation index, both gamma and Pearson Type III distributions) from an input precipitation dataset (in this case, the reduced resolution nClimGrid precipitation dataset provided in the example inputs directory). The input dataset is monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The index will be computed at 6-month and 12-month timescales. The output files will be /results/nclimgrid_lowres_spi_gamma_06.nc, /results/nclimgrid_lowres_spi_gamma_12.nc, /results/nclimgrid_lowres_spi_pearson_06.nc, and /results/nclimgrid_lowres_spi_pearson_12.nc.

SPEI monthly

$ python process_grid.py --index spei --periodicity monthly --netcdf_precip ../example_inputs/nclimgrid_lowres_prcp.nc --var_name_precip  prcp --netcdf_pet ../example_inputs/nclimgrid_lowres_pet.nc --var_name_pet pet --output_file_base ../results/nclimgrid_lowres --scales 9 18 --calibration_start_year 1951 --calibration_end_year 2010

The above command will compute SPEI (standardized precipitation evapotranspiration index, both gamma and Pearson Type III distributions) from input precipitation and potential evapotranspiration datasets (in this case, the reduced resolution nClimGrid precipitation and PET datasets provided in the example inputs directory). The input datasets are monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The index datasets will be computed at 9-month and 18-month timescales. The output files will be /results/nclimgrid_lowres_spi_gamma_09.nc, /results/nclimgrid_lowres_spi_gamma_18.nc, /results/nclimgrid_lowres_spi_pearson_09.nc, and /results/nclimgrid_lowres_spi_pearson_18.nc.

Palmers monthly

$ python process_grid.py --index palmers --periodicity monthly --netcdf_precip ../example_inputs/nclimgrid_lowres_prcp.nc --var_name_precip prcp --netcdf_pet ../example_inputs/nclimgrid_lowres_pet.nc --var_name_pet pet --netcdf_awc ../example_inputs/nclimgrid_lowres_soil.nc  --var_name_awc awc --output_file_base ../results/nclimgrid_lowres --calibration_start_year 1951 --calibration_end_year 2010

The above command will compute the Palmer drought indices: PDSI (original Palmer Drought Severity Index), PHDI (Palmer Hydrological Drought Index), PMDI (Palmer Modified Drought Index), Z-Index (Palmer Z-Index), and SCPDSI (Self-calibrated Palmer Drought Severity Index) from input precipitation, potential evapotranspiration, and available water capacity datasets (in this case, the reduced resolution nClimGrid precipitation, PET, and AWC datasets provided in the example inputs directory). The input datasets are monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The output files will be /results/nclimgrid_lowres_pdsi.nc, /results/nclimgrid_lowres_phdi.nc/results/nclimgrid_lowres_pmdi.nc, /results/nclimgrid_lowres_scpdsi.nc, and /results/nclimgrid_lowres_zindex.nc.