Core API

Documentation of the core API of pyaerocom.

Data classes

Gridded data

Ungridded data

Co-located data

Station data

Other data classes

Co-location routines

High-level co-location engine

Low-level co-location functions

Co-locating ungridded observations

Reading of gridded data

Gridded data specifies any dataset that can be represented and stored on a regular grid within a certain domain (e.g. lat, lon time), for instance, model output or level 3 satellite data, stored, for instance, as NetCDF files. In pyaerocom, the underlying data object is GriddedData and pyaerocom supports reading of such data for different file naming conventions.

Gridded data using AeroCom conventions

Gridded data using EMEP conventions

Reading of ungridded data

Other than gridded data, ungridded data represents data that is irregularly sampled in space and time, for instance, observations at different locations around the globe. Such data is represented in pyaerocom by UngriddedData which is essentially a point-cloud dataset. Reading of UngriddedData is typically specific for different observational data records, as they typically come in various data formats using various metadata conventions, which need to be harmonised, which is done during the data import.

The following flowchart illustrates the architecture of ungridded reading in pyaerocom. Below are information about the individual reading classes for each dataset (blue in flowchart), the abstract template base classes the reading classes are based on (dark green) and the factory class ReadUngridded (orange) which has registered all individual reading classes. The data classes that are returned by the reading class are indicated in light green.

ReadUngridded factory class

Factory class that has all reading class for the individual datasets registered.

ReadUngriddedBase template class

All ungridded reading routines are based on this template class.

AERONET

Aerosol Robotic Network (AERONET)

AERONET base class

All AERONET reading classes are based on the template ReadAeronetBase class which, in turn inherits from ReadUngriddedBase.

AERONET Sun (V3)

AERONET SDA (V3)

AERONET Inversion (V3)

EARLINET

European Aerosol Research Lidar Network (EARLINET)

EBAS

EBAS is a database with atmospheric measurement data hosted by the Norwegian Institute for Air Research. Declaration of AEROCOM variables in EBAS and assocaited information such as acceptable minimum and maximum values occurs in pyaerocom/data/variables.ini .

EBAS (low level)

EEA data

EEA base reader

Reader for European air pollution data from EEA AqERep files.

EEA E2a product (NRT)

Near realtime EEA data.

EEA E1a product (QC)

Quality controlled EEA data.

AirNow data

Reader for air quality measurements from North America.

MarcoPolo data

Reader for air quality measurements for China from the EU-FP7 project MarcoPolo.

GHOST

GHOST (Globally Harmonised Observational Surface Treatment) project developed at the Earth Sciences Department of the Barcelona Supercomputing Center (see e.g., Petetin et al., 2020 for more information).

Further I/O features

Note

The pyaerocom.io package also includes all relevant data import and reading routines. These are introduced above, in Section reading.

AeroCom database browser

File naming conventions

Iris helpers

Handling of cached ungridded data objects

I/O utils

I/O helpers

Metadata and vocabulary standards

Variables

Variable collection

Variable class

Variable helpers

Variable name info

Helpers for auxiliary variables

Variable categorisations

Regions and data filtering

Region class and helper functions

Region definitions

Region filter

Land / Sea masks

Time and frequencies

Handling of time frequencies

Temporal resampling

Global constants

Vertical coordinate support

Note

BETA: most functionality of this module is currently not implemented in any of the pyaerocom standard API.

Trends computation

Trends engine

Helper methods

Utility functions

Helpers

Mathematical helpers

Geodesic calculations and topography

Units and unit conversion

Units helpers in base package

Units helpers in io sub-package

Plotting / visualisation (sub package plot)

The pyaerocom.plot package contains algorithms related to data visualisation and plotting.

Plotting of maps

Plotting coordinates on maps

Scatter plots

Heatmap plots

Colors schemes

Plot helper functions

Configuration and global constants

Basic configuration class

Will be initiated on input and is accessible via pyaerocom.const.

Config defaults related to gridded data

Config details related to observations

Settings and helper methods / classes for I/O of obervation data

Note

Some settings like paths etc can be found in pyaerocom.config.py

class pyaerocom.obs_io.AuxInfoUngridded(data_id, vars_supported, aux_requires, aux_merge_how, aux_funs=None, aux_units=None)

MAX_VARS_PER_METHOD = 2

check_status()

Check if specifications are correct and consistent

Raises:

• ValueError – If one of the class attributes is invalid

• NotImplementedError – If computation method contains more than 2 variables / datasets

to_dict()

Dictionary representation of this object

Ignores any potential private attributes.

pyaerocom.obs_io.OBS_ALLOW_ALT_WAVELENGTHS = True

This boolean can be used to enable / disable the former (i.e. use available wavelengths of variable in a certain range around variable wavelength).

pyaerocom.obs_io.OBS_WAVELENGTH_TOL_NM = 10.0

Wavelength tolerance for observations if data for required wavelength is not available

class pyaerocom.obs_io.ObsVarCombi(obs_id, var_name)

Molar masses and related helpers

Access to minimal test dataset

Low-level helper classes and functions

Small helper utility functions for pyaerocom

class pyaerocom._lowlevel_helpers.AsciiFileLoc(default=None, assert_exists=False, auto_create=False, tooltip=None, logger=None)

create(value)

class pyaerocom._lowlevel_helpers.BrowseDict(*args, **kwargs)

Dictionary-like object with getattr and setattr options

Extended dictionary that supports dynamic value generation (i.e. if an assigned value is callable, it will be executed on demand).

ADD_GLOB = []

FORBIDDEN_KEYS = []

IGNORE_JSON = []

Keys to be ignored when converting to json

MAXLEN_KEYS = 100.0

SETTER_CONVERT = {}

import_from(other) → None

Import key value pairs from other object

Other than update() this method will silently ignore input keys that are not contained in this object.

Parameters:

other (dict or BrowseDict) – other dict-like object containing content to be updated.

Raises:

ValueError – If input is inalid type.

Return type:

None

items() → a set-like object providing a view on D's items

json_repr() → dict

Convert object to serializable json dict

Returns:

content of class

Return type:

dict

keys() → a set-like object providing a view on D's keys

pretty_str()

to_dict()

values() → an object providing a view on D's values

class pyaerocom._lowlevel_helpers.ConstrainedContainer(*args, **kwargs)

Restrictive dict-like class with fixed keys

This class enables to create dict-like objects that have a fixed set of keys and value types (once assigned). Optional values may be instantiated as None, in which case the first time instantiation definecs its type.

Note

The limitations for assignments are only restricted to setitem operations and attr assignment via “.” works like in every other class.

Example

class MyContainer(ConstrainedContainer):

def __init__(self):

self.val1 = 1 self.val2 = 2 self.option = None

>>> mc = MyContainer()
>>> mc['option'] = 42

CRASH_ON_INVALID = True

class pyaerocom._lowlevel_helpers.DictStrKeysListVals

validate(val: dict)

class pyaerocom._lowlevel_helpers.DictType

validate(val)

class pyaerocom._lowlevel_helpers.DirLoc(default=None, assert_exists=False, auto_create=False, tooltip=None, logger=None)

create(value)

class pyaerocom._lowlevel_helpers.EitherOf(allowed: list)

validate(val)

class pyaerocom._lowlevel_helpers.FlexList

list that can be instantated via input str, tuple or list or None

validate(val)

class pyaerocom._lowlevel_helpers.JSONFile(default=None, assert_exists=False, auto_create=False, tooltip=None, logger=None)

create(value)

validate(value)

class pyaerocom._lowlevel_helpers.ListOfStrings

validate(val)

class pyaerocom._lowlevel_helpers.Loc(default=None, assert_exists=False, auto_create=False, tooltip=None, logger=None)

Abstract descriptor representing a path location

Descriptor??? See here: https://docs.python.org/3/howto/descriptor.html#complete-practical-example

Note

• Child classes need to implement create()

• value is allowed to be None in which case no checks are performed

abstract create(value)

validate(value)

class pyaerocom._lowlevel_helpers.NestedContainer(*args, **kwargs)

keys_unnested() → list

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

class pyaerocom._lowlevel_helpers.StrType

validate(val)

class pyaerocom._lowlevel_helpers.StrWithDefault(default: str)

validate(val)

class pyaerocom._lowlevel_helpers.TypeValidator(type)

validate(val)

class pyaerocom._lowlevel_helpers.Validator

abstract validate(val)

pyaerocom._lowlevel_helpers.check_dir_access(path)

Uses multiprocessing approach to check if location can be accessed

Parameters:

loc (str) – path that is supposed to be checked

Returns:

True, if location is accessible, else False

Return type:

bool

pyaerocom._lowlevel_helpers.check_dirs_exist(*dirs, **add_dirs)

pyaerocom._lowlevel_helpers.check_make_json(fp, indent=4)

Make sure input json file exists

Parameters:

• fp (str) – filepath to be checked (must end with .json)

• indent (int) – indentation of json file

Raises:

ValueError – if filepath does not exist.

Returns:

input filepath.

Return type:

str

pyaerocom._lowlevel_helpers.check_write_access(path)

Check if input location provides write access

Parameters:

path (str) – directory to be tested

pyaerocom._lowlevel_helpers.chk_make_subdir(base, name)

Check if sub-directory exists in parent directory

pyaerocom._lowlevel_helpers.dict_to_str(dictionary, indent=0, ignore_null=False)

Custom function to convert dictionary into string (e.g. for print)

Parameters:

• dictionary (dict) – the dictionary

• indent (int) – indent of dictionary content

• ignore_null (bool) – if True, None entries in dictionary are ignored

Returns:

the modified input string

Return type:

str

pyaerocom._lowlevel_helpers.invalid_input_err_str(argname, argval, argopts)

Just a small helper to format an input error string for functions

Parameters:

• argname (str) – name of input argument

• argval – (invalid) value of input argument

• argopts – possible input args for arg

Returns:

formatted string that can be parsed to an Exception

Return type:

str

pyaerocom._lowlevel_helpers.list_to_shortstr(lst, indent=0)

Custom function to convert a list into a short string representation

pyaerocom._lowlevel_helpers.merge_dicts(dict1, dict2, discard_failing=True)

Merge two dictionaries

Parameters:

• dict1 (dict) – first dictionary

• dict2 (dict) – second dictionary

• discard_failing (bool) – if True, any key, value pair that cannot be merged from the 2nd into the first will be skipped, which means, the value of the output dict for that key will be the one of the first input dict. All keys that could not be merged can be accessed via key ‘merge_failed’ in output dict. If False, any Exceptions that may occur will be raised.

Returns:

merged dictionary

Return type:

dict

pyaerocom._lowlevel_helpers.read_json(file_path)

Read json file

Parameters:

file_path (str) – json file path

Returns:

content as dictionary

Return type:

dict

pyaerocom._lowlevel_helpers.round_floats(in_data, precision=5)

simple helper method to change all floats of a data structure to a given precision. For nested structures, this method is called recursively to go through all levels

Parameters:

in_data (float, dict, tuple, list) – data structure whose numbers should be limited in precision

Returns:

all the floats in in_data with limited precision tuples in the structure have been converted to lists to make them mutable

Return type:

in_data

pyaerocom._lowlevel_helpers.sort_dict_by_name(d, pref_list: list | None = None) → dict

Sort entries of input dictionary by their names and return ordered

Parameters:

• d (dict) – input dictionary

• pref_list (list, optional) – preferred order of items (may be subset of keys in input dict)

Returns:

sorted and ordered dictionary

Return type:

dict

pyaerocom._lowlevel_helpers.str_underline(title: str, indent: int = 0)

Create underlined string

pyaerocom._lowlevel_helpers.write_json(data_dict, file_path, **kwargs)

Save json file

Parameters:

• data_dict (dict) – dictionary that can be written to json file

• file_path (str) – output file path

• **kwargs – additional keyword args passed to simplejson.dumps() (e.g. indent, )

Custom exceptions

Module containing pyaerocom custom exceptions

exception pyaerocom.exceptions.AeroValConfigError

exception pyaerocom.exceptions.AeroValTrendsError

exception pyaerocom.exceptions.AeronetReadError

exception pyaerocom.exceptions.CacheReadError

exception pyaerocom.exceptions.CacheWriteError

exception pyaerocom.exceptions.CachingError

exception pyaerocom.exceptions.ColocationError

exception pyaerocom.exceptions.ColocationSetupError

exception pyaerocom.exceptions.CoordinateError

exception pyaerocom.exceptions.CoordinateNameError

exception pyaerocom.exceptions.DataCoverageError

exception pyaerocom.exceptions.DataDimensionError

exception pyaerocom.exceptions.DataExtractionError

exception pyaerocom.exceptions.DataIdError

exception pyaerocom.exceptions.DataQueryError

exception pyaerocom.exceptions.DataRetrievalError

exception pyaerocom.exceptions.DataSearchError

exception pyaerocom.exceptions.DataSourceError

exception pyaerocom.exceptions.DataUnitError

exception pyaerocom.exceptions.DeprecationError

exception pyaerocom.exceptions.DimensionOrderError

exception pyaerocom.exceptions.EEAv2FileError

exception pyaerocom.exceptions.EbasFileError

exception pyaerocom.exceptions.EntryNotAvailable

exception pyaerocom.exceptions.EvalEntryNameError

exception pyaerocom.exceptions.FileConventionError

exception pyaerocom.exceptions.InitialisationError

exception pyaerocom.exceptions.LongitudeConstraintError

exception pyaerocom.exceptions.MetaDataError

exception pyaerocom.exceptions.NasaAmesReadError

exception pyaerocom.exceptions.NetcdfError

exception pyaerocom.exceptions.NetworkNotImplemented

exception pyaerocom.exceptions.NetworkNotSupported

exception pyaerocom.exceptions.NotInFileError

exception pyaerocom.exceptions.ResamplingError

exception pyaerocom.exceptions.StationCoordinateError

exception pyaerocom.exceptions.StationNotFoundError

exception pyaerocom.exceptions.TemporalResolutionError

exception pyaerocom.exceptions.TemporalSamplingError

exception pyaerocom.exceptions.TimeMatchError

exception pyaerocom.exceptions.TimeZoneError

exception pyaerocom.exceptions.UnitConversionError

exception pyaerocom.exceptions.UnknownRegion

exception pyaerocom.exceptions.UnresolvableTimeDefinitionError

Is raised if time definition in NetCDF file is wrong and cannot be corrected

exception pyaerocom.exceptions.VarNotAvailableError

exception pyaerocom.exceptions.VariableDefinitionError

exception pyaerocom.exceptions.VariableNotFoundError