Adding Custom Variables with EMEP

In this tutorial we will go through how to add custom variables to our PyAeroVal evaluation, as well as how to read these from EMEP. We will add two new variables: elementalcarbon and organiccarbon. These variables are just meant as illustrative examples, and variables for elemental and organic carbon do already exist in Pyaerocom.

Adding the variables will be done in three steps:

  1. Register the new variables in Pyaerocom

  2. Tell Pyaercom how to read organiccarbon from EMEP

  3. Combine two existing variables to make elementalcarbon

Before continuing, please note: 1. You need to have access to PPI/Lustre to do this tutorial, due to it using EMEP files found there. 2. Adding custom variables should only be done for testing, and not production! See the last section.

Registering the New Variables

Here we will look at how to add an extra variable, in the same way as is done in variables.ini. This is needed so that PyAerocom knows about the variable and it units.

PyAerocom has a static object called config, which contains a function for adding new variables

[2]:

from pyaerocom.variable import Variable
from pyaerocom import const

variables = {
    "elemental_carbon": Variable(var_name="elementalcarbon", units="ug Cm-3"),
    "organic_carbon": Variable(var_name="organiccarbon", units="ug Cm-3"),
}

const.register_custom_variables(variables)

The function register_custom_variables takes a dictionary where the values are Variable objects. These objects need at least the variable name and unit. It can also take other arguments, such as min and max. In `variables.ini <https://github.com/metno/pyaerocom/blob/main-dev/pyaerocom/data/variables.ini>`__ you can see all the other arguments that are used.

Reading From EMEP

PyAerocom now know that we have two new variables, but since they are not define anywhere inside the PyAerocom code, PyAerocom does not know from where to read the data. We will here tell PyAerocom how to read the two variables from EMEP.

Reading From an EMEP File

The most common way of reading data from EMEP is to just read them directly from an EMEP file. To do this, PyAerocom needs to know what that variable is called in EMEP. Normally PyAerocom gets this mapping from this file, but since we just introduced a new variables, we need to define this mapping our self. This is done when defining our EMEP model entry in our config by adding the option emep_vars

[3]:

folder_EMEP = "/lustre/storeB/project/fou/kl/emep/ModelRuns/2021_REPORTING/TRENDS/2018"

EMEP = dict(
        model_id="EMEP",
        model_data_dir=folder_EMEP,
        gridded_reader_id={"model": "ReadMscwCtm"}, # Tells pyaerocom to use the EMEP reader instead of the default aerocom reader

        emep_vars={"organiccarbon": "SURF_ugC_PM_OM25"},
    )

Our new mapping is thus defined as "organiccarbon": "SURF_ugC_PM_OM25". PyAerocom will now read our first variables

Combining Existing Variables

The other way PyAerocom gets variables from EMEP is to combine other variables. In this case we will combine the already existing variables concecCoarse and concecFine to make our new variable elementalcarbon.

This takes a bit more work than what we did above. We will start by creating a new function we will use to tell PyAerocom how to combine concecCoarse and concecFine

[4]:

def calc_elecmentalcarbon(concecCoarse, concecFine):

    elementalcarbon = concecCoarse.copy(deep=True) + concecFine.copy(
        deep=True
    )  # Adds the two variables
    elementalcarbon.attrs["units"] = "ug C m-3"  # Make sure the unit is correct
    return elementalcarbon

This function tells PyAerocom that we want to add the two variables. It also makes sure that we have the correct unit (in this case the wrong unit, but again this is just an example). We now need to tell PyAerocom that we have this function. This is again done in the EMEP model entry

[5]:

folder_EMEP = "/lustre/storeB/project/fou/kl/emep/ModelRuns/2021_REPORTING/TRENDS/2018"

EMEP = dict(
    model_id="EMEP",
    model_data_dir=folder_EMEP,
    gridded_reader_id={
        "model": "ReadMscwCtm"
    },  # Tells pyaerocom to use the EMEP reader instead of the default aerocom reader
    emep_vars={"organiccarbon": "SURF_ugC_PM_OM25"},
    model_read_aux={
        "elementalcarbon": {
            "vars_required": ["concecCoarse", "concecFine"],
            "fun": calc_elecmentalcarbon,
        }
    },
)

We see here that we have the new option model_read_aux. This option takes keys which are our new variables. The values are also dictionaries:

  • vars_required: A list of the PyAerocom variables needed for the calculation. In our case concecCoarse and concecFine.

  • fun: The function which tells PyAerocom how to combine the variables. In our case the function we made just above

Note that for other used vars_required doesn’t need to have two variables in the list. We can make combinations with arbitrary many variables. We can also use newly defined variables, such as organiccarbon.

The config for this evaluation can be found here.

Best Practice

Note that this way of adding custom variables should only be used for testing, and not in production. Meaning that when you have found all the variables you want to add to AeroTools (Pyaerocom and/or the EMEP reader), you should make an issue on the PyAerocom github or ask one of the developers to add it. There are couple of reasons why

  1. It’s it not as easy to reproduce experiments where you have added custom variables, since your additions are not documented on the webpage or in the PyAerocom code.

  2. You might get errors/memory errors which are difficult to handle, especially with the EMEP reader

  3. Other parts having to do with the variables are hardcoded in PyAerocom/PyAeroval. Examples are axis scaling, display names, etc.