Climate and Drought Indices in Python

This project contains Python implementations of various climate index algorithms which provide a geographical and temporal picture of the severity of precipitation and temperature anomalies useful for climate monitoring and research.  This code is currently considered ‘Beta’ as NIDIS performs additional testing and verification.  If you are currently running another implementation of the algorithms, how do these Python indices compare?  Please help us by reporting feedback to nidis.questions@noaa.gov.

The following indices are provided:

• SPI, Standardized Precipitation Index, utilizing both gamma and Pearson Type III distributions
• SPEI, Standardized Precipitation Evapotranspiration Index, utilizing both gamma and Pearson Type III distributions
• PET, Potential Evapotranspiration, utilizing either Thornthwaite or Hargreaves equations
• PDSI, Palmer Drought Severity Index
• scPDSI, Self-calibrated Palmer Drought Severity Index
• PHDI, Palmer Hydrological Drought Index
• Z-Index, Palmer moisture anomaly index (Z-index)
• PMDI, Palmer Modified Drought Index
• PNP, Percentage of Normal Precipitation

This Python implementation of the above climate index algorithms is being developed with the following goals in mind:

• to provide an open source software package to compute a suite of climate indices commonly used for climate monitoring, with well documented code that is faithful to the relevant literature and which produces scientifically verifiable results
• to provide a central, open location for participation and collaboration for researchers, developers, and users of climate indices
• to facilitate standardization and consensus on best-of-breed climate index algorithms and corresponding compliant implementations in Python
• to provide transparency into the operational code used for climate monitoring activities at NCEI/NOAA, and consequent reproducibility of published datasets computed from this package
• to incorporate modern software engineering principles and programming best practices

Getting started

The installation and configuration described below is performed using a bash shell, either on Linux, Windows, or MacOS.

Windows users will need to install and configure a bash shell in order to follow the usage shown below. We recommended either babun or Cygwin for this purpose.

Download the code

Download the zip file from the links at the top of this page.  Unzip and move into the source directory.

$ cd climate_indices

Within this directory, there are six subdirectories:

• climate_indices: main computational package
• tests: unit tests for the main package
• scripts: scripts and supporting utility modules used to perform processing of indices computations on climatological datasets (typically grids or US climate division datasets in NetCDF)
• example_inputs: example/reference datasets that can be used as inputs to the processing scripts
• notebooks: Jupyter Notebooks describing the internals of the computational modules
• docs: documentation files
• results: empty output directory for example output files
• extra_scripts: other scripts that allow extra functions including data ingest, comparisons scripts for results...etc.

Configure the Python environment

This project’s code is written in Python 3. It’s recommended to use either the Miniconda3 (minimal Anaconda) orAnaconda3 distribution. The below instructions will be Anaconda specific (although relevant to any Python virtualenv ), and assume the use of a bash shell.

A new Anaconda environment can be created using the conda environment management system that comes packaged with Anaconda. In the following examples, we’ll use an environment named indices_env (any environment name can be used instead of indices_env) which will be created and populated with all required dependencies through the use of the provided setup.py file.

First create the Python environment:

$ conda create -n indices_env

The environment created can now be ‘activated’:

$ source activate indices_env

Once the environment has been activated then subsequent Python commands will run in this environment where the package dependencies for this project are present.

Now the package can be added to the environment along with all required modules (dependencies) via python and the setup.py script:

$ python setup.py install

Testing

Initially, all tests should be run for validation:

$ export NUMBA_DISABLE_JIT=1

$ python setup.py test

$ unset NUMBA_DISABLE_JIT

If you run the above from the main branch and get an error then please send a report and/or add an issue, as all tests should pass.

The Numba environment variable is set/unset in order to bypass Numba’s just-in-time compilation process, which significantly reduces testing times.

Indices Processing

Included are scripts which interact with the core computational package to compute one or more climate indices. These are process_grid.py which is used to compute indices from gridded NetCDF datasets, and process_divisions.py which is used to compute indices from US climate division NetCDF datasets. Both of these scripts are located within the “scripts” directory in the zipped file at the top of this page.

These Python scripts are written to be run via bash shell commands, i.e.

$ python process_grid.py <options>

Input Data Requirements

This software enables the user to calculate several drought indices using any gridded NetCDF dataset. However, there are certain specifications for input files that vary based on input type:
 

Time-series Dimension Specifics:
      When relevant, the time dimension should include full years (padded with NaN or Null values) and leap years with 366 days for daily files.

Example Command Line Invocations

Example Output files

All example output files will be in a netCDF file format following the Climate and Forecast metadata conventions. Each output file will have three dimensions, time, longitude, and latitude. The “results” directory included in the examples below serves as the output directory and will be empty until an example is properly conducted. These netCDF files can be opened and visualized with several publicly available software including NOAA’s Weather and Climate Toolkit and NASA Goddard’s Panoply. For the SPI and SPEI examples below there will be periods of no-data matching the supplied time-scales used, due to the nature of the algorithms used. For example, a 3-month SPI starting on January 1^st requires 3-months of data to create a statistically valid SPI value, so the data values will not start until April 1^st. The output files do not trim the initial supplied time-scale, the first 3-months in the example above, of no-data. The length of the time dimension will still match that of the input data.