Data Request

Data Request Structure

 A number of model intercomparison projects (MIPs) have been set up for the various scientific objectives of CMIP6. These can request data from various experiments (Data Request). Depending on the classification of the experiment ("tier"), such a request includes various variables which are also provided with a classification ("priority"). The variable defined by the MIPs is called "MIP variable", whereas their programmatic implementation(s) are referred to as "CMOR variables", in accordance with the CMOR program (Climate Model Output Rewriter, see below for details) used to reformat and adapt the model data to the CMIP standard.

The CMIP6 Data Request is published in the form of two xml files. However, any software, including that of this project, has to be aware that changes may become necessary. The changes may occur due to continuing adaptation of the request to requirements of the participants (MIPs and modellers).

The "Data Request Python API" (DreqPy API)

In order to support the scientists in generating the requested data, Martin Juckes of the BADC (British Atmospheric Data Center) has developed the DataRequest Python API (DreqPy), an interface written in the Python programming language, allowing to create customized request depending on the choice of experiments to be conducted as well as on the MIPs that shall be supported. In addition the data request depends on the chosen experiment "tier" and variable "priority".  Fiddeling around with these parameters the volume of the data request may be adapted to the available storage capacities.

The Data Request output format is available as an Excel table or as a comma-separated variable list. It can be searched yet interactively by means of various classes and functions available in DreqPy.
For the interactive approach, CEDA also hosts a web page which allows browsing the variables (and other objects like MIPs, scientific objectives, etc.).

Furthermore, it is possible to obtain an approximate data volume estimate of a data request for a given grid.

Data Request Web GUI

A WebGUI has been developed at the DKRZ in order to facilitate the use and configuration of DreqPy. Thereby no local installation of the software is needed. A customized data request may be generated by simply checking some choices, and the final product is available for download in .csv or .xlsx format. It is also possible to calculate the volume estimate for this customized Data Request.

Furthermore, it is possible to use the resulting data request (in .csv format) in order to control the data formatting into the CMIP standard (see section below).

A short video demonstrating the usage of the Data Request Web GUI is available. The video and further documentation can be accessed as well through the WebGUI using the tab 'Documentation&Links'.

Variable Mapping Web GUI

Variables are usually named differently within climate models than defined by the standard. Therefore a mapping between model output variables and requested variables is needed.
For this purpose the WebGUI has been extended, building on previous efforts of Karl-Hermann Wieners (MPI-M). The web application allows for simultaneous use by multiple scientists. Space is provided to exchange comments and questions.

To make sure that always the latest variable definitions and data request information is used, the official DreqPy software (as discussed above) is integrated into this web application. The subsequent video gives an idea on how to use the Variable Mapping Web GUI. The video, a slideshow, and further documentation can also be accessed from the WebGUI (Tab: Documentation & Links).

Using Data Request Specifications for Data Post Processing

The information entered in the Variable Mapping WebGUI is used to support the writing of scripts for standard compliant data processing.

In order to transform raw model output into project variables the cdo operator 'expr' is used as far as possible. The final re-writing of the variables (CMOR=ClimateModelOutputRewriter) is done with the newly developed cdo operator 'cmor', which calls the CMOR3 library developed at the PCMDI (Program for Climate Model Diagnosis and Intercomparison).