Checking pyaerocom installation and access to data๏ƒ

Import pyaerocom๏ƒ

It all begins with an import:

[2]:

import pyaerocom as pya
import logging
logging.getLogger().setLevel(logging.ERROR) # Set level in information outputted by pyaerocom to the console.\n",
pya.__version__

[2]:

'0.21.0.dev0'

When imported, pyaerocom will automatically check access to different default data locations (e.g. mount to PPI at MET Norway, or ~/MyPyaerocom/data), and in case a data source location is detected, associated data directories for accessing these data are instantiated (details below).

Pyaerocom also comes equipped with a command line interface (CLI).

[10]:

! pya --help

                                                                                
 Usage: pya [OPTIONS] COMMAND [ARGS]...                                         
                                                                                
 ๐Ÿฆ„ Pyaerocom Command Line Interface

โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ --version             -V                                                     โ”‚
โ”‚ --install-completion            Install completion for the current shell.    โ”‚
โ”‚ --show-completion               Show completion for the current shell, to    โ”‚
โ”‚                                 copy it or customize the installation.       โ”‚
โ”‚ --help                          Show this message and exit.                  โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
โ•ญโ”€ Commands โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ aeroval         Run an AeroVal experiment as described in a json config file โ”‚
โ”‚ browse          Browse database (e.g., browse <DATABASE>)                    โ”‚
โ”‚ clearcache      Delete cached data objects                                   โ”‚
โ”‚ getsampledata   Downloads a minimal sample dataset.                          โ”‚
โ”‚ listcache       List cached data objects                                     โ”‚
โ”‚ ppiaccess       Check if MET Norway's PPI can be accessed                    โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

Check available datasets and directories๏ƒ

Accessible data and default paths for certain datasets are available (and can be updated via the const module).

[3]:

pya.const

[3]:

<pyaerocom.config.Config at 0x76fc682c95d0>

Data search directories๏ƒ

[4]:

pya.const.DATA_SEARCH_DIRS

[4]:

['/lustre/storeB/project/aerocom/aerocom1/',
 '/lustre/storeB/project/aerocom/aerocom2/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/CMIP6',
 '/lustre/storeB/project/aerocom/aerocom-users-database/DOMOS',
 '/lustre/storeB/project/aerocom/aerocom-users-database/C3S-Aerosol',
 '/lustre/storeB/project/aerocom/aerocom-users-database/ECLIPSE',
 '/lustre/storeB/project/aerocom/aerocom-users-database/SATELLITE-DATA/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/CCI-Aerosol/CCI_AEROSOL_Phase2/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/ACCMIP/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/ECMWF/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP_COPERNICUS/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP_GLOBAL/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP_SVN_TEST/',
 '/lustre/storeB/project/aerocom/aerocom2/NorESM_SVN_TEST/',
 '/lustre/storeB/project/aerocom/aerocom2/INCA/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-II/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-I/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III-2019/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III-Trend/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/CCI-Aerosol/CCI_AEROSOL_Phase1/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II-IND3/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II-IND2/',
 '/lustre/storeB/project/fou/kl/CAMS61/',
 '/lustre/storeB/project/aerocom/aerocom1/AEROCOM_OBSDATA/PYAEROCOM/',
 '/lustre/storeB/project/fou/kl/CAMS2_40/task4041/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/DOMOS/']

[27]:

! pya getsampledata

Extracting 'testdata-minimal' from '/home/lewisb/MyPyaerocom/testdata-minimal.tar.gz.20231116' to '/home/lewisb/Projects/pyaerocom-tutorials/data'

When I run this we can see multiple data sources, but you might not see any. That is because I already have access to alot of data, but you, being new to this, donโ€™t have any. This means that we need some data we all can use!

Downloading the pyaerocom testdata-minimal dataset๏ƒ

The testdata-minimal dataset was developed for automatic testing of pyaerocom and is well suited to illustrate the main features of pyaerocom, without too requiring too heavy computing resources or data storage. It is very easy to get these data:

[ ]:

%%bash

pya getsampledata

Now we have a path, where there is supposed to be some data. Awesomeness!

[ ]:

dataloc = f'./data/testdata-minimal/'
dataloc

Side comment: If this way of formatting python strings looks weird to you, donโ€™t worry, this is because it is a rather new feature (as of Nov 2020).

[12]:

import os
os.listdir(dataloc)

[12]:

['obsdata', 'config', 'modeldata', 'coldata', 'scripts', 'README.md']

Letโ€™s look into the modeldata directory (obsdata follows later).

[13]:

os.listdir(dataloc + 'modeldata')

[13]:

['TM5-met2010_CTRL-TEST', 'EMEP_2017']

Adding data search directories๏ƒ

Great, found something. Letโ€™s tell pyaerocom about it.

[ ]:

pya.const.add_data_search_dir(dataloc + 'modeldata')

Now, letโ€™s repeat what we did before.

[15]:

pya.const.DATA_SEARCH_DIRS

[15]:

['/lustre/storeB/project/aerocom/aerocom1/',
 '/lustre/storeB/project/aerocom/aerocom2/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/CMIP6',
 '/lustre/storeB/project/aerocom/aerocom-users-database/DOMOS',
 '/lustre/storeB/project/aerocom/aerocom-users-database/C3S-Aerosol',
 '/lustre/storeB/project/aerocom/aerocom-users-database/ECLIPSE',
 '/lustre/storeB/project/aerocom/aerocom-users-database/SATELLITE-DATA/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/CCI-Aerosol/CCI_AEROSOL_Phase2/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/ACCMIP/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/ECMWF/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP_COPERNICUS/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP_GLOBAL/',
 '/lustre/storeB/project/aerocom/aerocom2/EMEP_SVN_TEST/',
 '/lustre/storeB/project/aerocom/aerocom2/NorESM_SVN_TEST/',
 '/lustre/storeB/project/aerocom/aerocom2/INCA/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-II/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-I/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III-2019/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III-Trend/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/CCI-Aerosol/CCI_AEROSOL_Phase1/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II-IND3/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II-IND2/',
 '/lustre/storeB/project/fou/kl/CAMS61/',
 '/lustre/storeB/project/aerocom/aerocom1/AEROCOM_OBSDATA/PYAEROCOM/',
 '/lustre/storeB/project/fou/kl/CAMS2_40/task4041/',
 '/lustre/storeB/project/aerocom/aerocom-users-database/DOMOS/',
 '/home/lewisb/MyPyaerocom/testdata-minimal/modeldata']

[18]:

! pya browse *TM5*

Searching database for matches of *TM5*

Pyaerocom ReadGridded
---------------------
Data ID: TM5JRCCY2IPCCV1_SR6SA
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/TM5JRCCY2IPCCV1_SR6SA/renamed
Available experiments: ['SR6SA']
Available years: [2001]
Available frequencies ['monthly']
Available variables: ['MMR_BCSR6SA', 'MMR_NO3SR6SA', 'MMR_POMSR6SA', 'MMR_SO4SR6SA']

Pyaerocom ReadGridded
---------------------
Data ID: TM5-JRC-cy2-ipcc-v1_SR1
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/TM5-JRC-cy2-ipcc-v1_SR1/renamed
Available experiments: ['SR1']
Available years: [2001]
Available frequencies ['monthly']
Available variables: ['vmro3']

Pyaerocom ReadGridded
---------------------
Data ID: TM5JRCCY2IPCCV1_SR6EU
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/TM5JRCCY2IPCCV1_SR6EU/renamed
Available experiments: ['SR6EU']
Available years: [2001]
Available frequencies ['monthly']
Available variables: ['MMR_BCSR6EU', 'MMR_NO3SR6EU', 'MMR_POMSR6EU', 'MMR_SO4SR6EU']

Pyaerocom ReadGridded
---------------------
Data ID: TM5JRCCY2IPCCV1_SR1
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/TM5JRCCY2IPCCV1_SR1/renamed
Available experiments: ['SR1']
Available years: [2001]
Available frequencies ['monthly']
Available variables: ['SCONCBC', 'SCONCNO3', 'SCONCPM25', 'SCONCPOM', 'SCONCSO4']

Pyaerocom ReadGridded
---------------------
Data ID: TM5JRCCY2IPCCV1_SR6NA
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/TM5JRCCY2IPCCV1_SR6NA/renamed
Available experiments: ['SR6NA']
Available years: [2001]
Available frequencies ['monthly']
Available variables: ['MMR_BCSR6NA', 'MMR_NO3SR6NA', 'MMR_POMSR6NA', 'MMR_SO4SR6NA']

Pyaerocom ReadGridded
---------------------
Data ID: TM5JRCCY2IPCCV1_SR6EA
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/HTAP-PHASE-I/TM5JRCCY2IPCCV1_SR6EA/renamed
Available experiments: ['SR6EA']
Available years: [2001]
Available frequencies ['monthly']
Available variables: ['MMR_BCSR6EA', 'MMR_NO3SR6EA', 'MMR_POMSR6EA', 'MMR_SO4SR6EA']
Reading failed for TM5_B. Error: AttributeError("'NoneType' object has no attribute 'experiment'")

Pyaerocom ReadGridded
---------------------
Data ID: TM5-V3.A2.PRE
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II/TM5-V3.A2.PRE/renamed
Available experiments: ['']
Available years: [1850]
Available frequencies ['daily' 'monthly']
Available variables: ['abs550aer', 'abs550dryaer', 'airmass', 'asyaer', 'clt', 'drybc', 'drydms', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'drynoy', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec550aer', 'ec550dryaer', 'emibc', 'emidms', 'emidust', 'eminh3', 'eminox', 'emioa', 'emiso2', 'emiso4', 'emiss', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'precip', 'pressure', 'ps', 'rsds', 'rsdscs', 'rsdscsdif', 'rsdscsvis', 'rsdt', 'rsus', 'rsut', 'rsutcs', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'temp', 'vmrdms', 'vmrhno3', 'vmrno', 'vmrno2', 'vmrpan', 'vmrso2', 'wet3Dbc', 'wet3Ddu', 'wet3Dhno3', 'wet3Dnh4', 'wet3Dnoy', 'wet3Doa', 'wet3Dso2', 'wet3Dso4', 'wet3Dss', 'wetbc', 'wetdms', 'wetdust', 'wethno3', 'wetnh4', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'ang4487aer', 'od550gt1aer', 'fmf550aer', 'concNhno3', 'wetdu']

Pyaerocom ReadGridded
---------------------
Data ID: TM5-V3.A2.HCA-0
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II/TM5-V3.A2.HCA-0/renamed
Available experiments: ['']
Available years: [2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009]
Available frequencies ['daily' 'monthly']
Available variables: ['abs550aer', 'abs550dryaer', 'airmass', 'asyaer', 'drydms', 'drydust', 'dryso2', 'dryso4', 'dryss', 'ec550aer', 'ec550dryaer', 'emibc', 'emidms', 'emidust', 'emioa', 'emiso2', 'emiso4', 'emiss', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'pmid3d', 'precip', 'pressure', 'ps', 'scatc550dryaer', 'sconcbc', 'sconcdust', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'temp', 'wetbc', 'wetdms', 'wetdust', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'ang4487aer', 'od550gt1aer', 'fmf550aer', 'pmid']

Pyaerocom ReadGridded
---------------------
Data ID: TM5-V3.A2.CTRL
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II/TM5-V3.A2.CTRL/renamed
Available experiments: ['']
Available years: [2006]
Available frequencies ['daily' 'monthly' 'hourly']
Available variables: ['abs550aer', 'abs550dry1Daer', 'abs550dryaer', 'airmass', 'ang4487aer', 'asyaer', 'conccn1Dmode01', 'conccn1Dmode02', 'conccn1Dmode03', 'conccn1Dmode04', 'conccn1Dmode05', 'conccn1Dmode06', 'conccn1Dmode07', 'conccnmode01', 'conccnmode02', 'conccnmode03', 'conccnmode04', 'conccnmode05', 'conccnmode06', 'conccnmode07', 'drybc', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'drynoy', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec550aer', 'ec550dry1Daer', 'ec550dryaer', 'emibc', 'emidms', 'emidust', 'eminh3', 'eminox', 'emioa', 'emiso2', 'emiso4', 'emiss', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'mmr1Daerh2o', 'mmr1Dtr01', 'mmr1Dtr02', 'mmr1Dtr03', 'mmr1Dtr04', 'mmr1Dtr05', 'mmr1Dtr06', 'mmr1Dtr07', 'mmr1Dtr08', 'mmr1Dtr09', 'mmr1Dtr10', 'mmr1Dtr11', 'mmr1Dtr12', 'mmr1Dtr13', 'mmr1Dtr14', 'mmr1Dtr15', 'mmr1Dtr16', 'mmr1Dtr17', 'mmr1Dtr18', 'mmr1Dtr19', 'mmraerh2o', 'mmrbc', 'mmrdu', 'mmrno3', 'mmroa', 'mmrso4', 'mmrss', 'mmrtr01', 'mmrtr02', 'mmrtr03', 'mmrtr04', 'mmrtr05', 'mmrtr06', 'mmrtr07', 'mmrtr08', 'mmrtr09', 'mmrtr10', 'mmrtr11', 'mmrtr12', 'mmrtr13', 'mmrtr14', 'mmrtr15', 'mmrtr16', 'mmrtr17', 'mmrtr18', 'mmrtr19', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'pmid3d', 'precip', 'pressure', 'ps', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'temp', 'vmrdms', 'vmrhno3', 'vmrno', 'vmrno2', 'vmrpan', 'vmrso2', 'wet3Dbc', 'wet3Ddu', 'wet3Dhno3', 'wet3Dnh4', 'wet3Dnoy', 'wet3Doa', 'wet3Dso2', 'wet3Dso4', 'wet3Dss', 'wetbc', 'wetdust', 'wethno3', 'wetnh4', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'od550gt1aer', 'fmf550aer', 'concNhno3', 'pmid', 'wetdu']

Pyaerocom ReadGridded
---------------------
Data ID: TM5-V3.A2.HCA-IPCC
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-II/TM5-V3.A2.HCA-IPCC/renamed
Available experiments: ['']
Available years: [2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009]
Available frequencies ['daily' 'monthly' 'hourly']
Available variables: ['abs550aer', 'abs550dry1Daer', 'abs550dryaer', 'airmass', 'asyaer', 'clt', 'conccn1Dmode01', 'conccn1Dmode02', 'conccn1Dmode03', 'conccn1Dmode04', 'conccn1Dmode05', 'conccn1Dmode06', 'conccn1Dmode07', 'conccnmode01', 'conccnmode02', 'conccnmode03', 'conccnmode04', 'conccnmode05', 'conccnmode06', 'conccnmode07', 'drybc', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'drynoy', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec550aer', 'ec550dry1Daer', 'ec550dryaer', 'emibc', 'emidms', 'emidust', 'eminh3', 'eminox', 'emioa', 'emiso2', 'emiso4', 'emiss', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'mmr1Daerh2o', 'mmr1Dtr01', 'mmr1Dtr02', 'mmr1Dtr03', 'mmr1Dtr04', 'mmr1Dtr05', 'mmr1Dtr06', 'mmr1Dtr07', 'mmr1Dtr08', 'mmr1Dtr09', 'mmr1Dtr10', 'mmr1Dtr11', 'mmr1Dtr12', 'mmr1Dtr13', 'mmr1Dtr14', 'mmr1Dtr15', 'mmr1Dtr16', 'mmr1Dtr17', 'mmr1Dtr18', 'mmr1Dtr19', 'mmraerh2o', 'mmrbc', 'mmrdu', 'mmrno3', 'mmroa', 'mmrso4', 'mmrss', 'mmrtr01', 'mmrtr02', 'mmrtr03', 'mmrtr04', 'mmrtr05', 'mmrtr06', 'mmrtr07', 'mmrtr08', 'mmrtr09', 'mmrtr10', 'mmrtr11', 'mmrtr12', 'mmrtr13', 'mmrtr14', 'mmrtr15', 'mmrtr16', 'mmrtr17', 'mmrtr18', 'mmrtr19', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'pmid3d', 'precip', 'pressure', 'ps', 'rsds', 'rsdscs', 'rsdscsdif', 'rsdscsvis', 'rsdt', 'rsus', 'rsut', 'rsutcs', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'temp', 'vmrdms', 'vmrhno3', 'vmrno', 'vmrno2', 'vmrpan', 'vmrso2', 'wet3Dbc', 'wet3Ddu', 'wet3Dhno3', 'wet3Dnh4', 'wet3Dnoy', 'wet3Doa', 'wet3Dso2', 'wet3Dso4', 'wet3Dss', 'wetbc', 'wetdust', 'wethno3', 'wetnh4', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'ang4487aer', 'od550gt1aer', 'fmf550aer', 'concNhno3', 'pmid', 'wetdu']

Pyaerocom ReadGridded
---------------------
Data ID: TM5_AP3-CTRL2016
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III/TM5_AP3-CTRL2016/renamed
Available experiments: ['AP3']
Available years: [2006, 2008, 2010]
Available frequencies ['monthly' '3hourly']
Available variables: ['abs350aer', 'abs440aer', 'abs440dryaer', 'abs550aer', 'abs550dryaer', 'abs550drylt1aer', 'abs870aer', 'abs870dryaer', 'airmass', 'asyaer', 'asydryaer', 'deltaz3d', 'depbc', 'depdms', 'depdust', 'dephno3', 'depmsa', 'depn', 'depnh3', 'depnh4', 'depnhx', 'depno2', 'depno3', 'depnoy', 'depo3', 'depoa', 'deps', 'depso2', 'depso4', 'depss', 'dh', 'drybc', 'drydms', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'dryno3', 'drynoy', 'dryo3', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec440dryaer', 'ec550aer', 'ec550dryaer', 'ec550drylt1aer', 'ec870dryaer', 'emibc', 'emico', 'emidms', 'emidust', 'emiisop', 'emin', 'eminh3', 'eminox', 'emioa', 'emis', 'emiso2', 'emiso4', 'emiss', 'emiterp', 'humidity3d', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'od350aer', 'od440aer', 'od550aer', 'od550aer3d', 'od550aerh2o', 'od550bc', 'od550dryaer', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550lt1ss', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'pr', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcnh4', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'ta', 'temp', 'vmrch4', 'vmrco', 'vmrno', 'vmrno2', 'vmro3', 'vmroh', 'wetbc', 'wetdms', 'wetdust', 'wethno3', 'wetmsa', 'wetnh3', 'wetnh4', 'wetno3', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'ang4487aer', 'angabs4487aer', 'od550gt1aer', 'vmrox', 'fmf550aer', 'concNnh4', 'deltaz', 'humidity']

Pyaerocom ReadGridded
---------------------
Data ID: TM5_AP3-INSITU
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III/TM5_AP3-INSITU/renamed
Available experiments: ['AP3']
Available years: [2010]
Available frequencies ['monthly' 'daily' 'hourly']
Available variables: ['abs350aer', 'abs440aer', 'abs440dryaer', 'abs550aer', 'abs550dryaer', 'abs550drylt1aer', 'abs870aer', 'abs870dryaer', 'airmass', 'asyaer', 'asydryaer', 'depbc', 'depdms', 'depdust', 'dephno3', 'depmsa', 'depn', 'depnh3', 'depnh4', 'depnhx', 'depno2', 'depno3', 'depnoy', 'depo3', 'depoa', 'deps', 'depso2', 'depso4', 'depss', 'dh', 'drybc', 'drydms', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'dryno3', 'drynoy', 'dryo3', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec440dryaer', 'ec550aer', 'ec550dryaer', 'ec550drylt1aer', 'ec870dryaer', 'emibc', 'emico', 'emidms', 'emidust', 'emiisop', 'emin', 'eminh3', 'eminox', 'emioa', 'emis', 'emiso2', 'emiso4', 'emiss', 'emiterp', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'mmrbc', 'mmrdust', 'mmrmsa', 'mmrnh4', 'mmrno3', 'mmroa', 'mmrso4', 'mmrss', 'od350aer', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550lt1ss', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'plev', 'pr', 'precip', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'ta', 'temp', 'vmrch4', 'vmrco', 'vmrno', 'vmrno2', 'vmro3', 'vmroh', 'wetbc', 'wetdms', 'wetdust', 'wethno3', 'wetmsa', 'wetnh3', 'wetnh4', 'wetno3', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'ang4487aer', 'angabs4487aer', 'od550gt1aer', 'vmrox', 'fmf550aer']

Pyaerocom ReadGridded
---------------------
Data ID: TM5_AP3-CTRL2015
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III/TM5_AP3-CTRL2015/renamed
Available experiments: ['AP3']
Available years: [2010]
Available frequencies ['monthly']
Available variables: ['depbc', 'depdust', 'depno3', 'depoa', 'depso4', 'depss', 'drybc', 'drydust', 'dryno3', 'dryoa', 'dryso4', 'dryss', 'emibc', 'emidms', 'emidust', 'eminox', 'emioa', 'emiso2', 'emiso4', 'emiss', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'od550aer', 'od550bc', 'od550dust', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'sconcbc', 'sconcdust', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'wetbc', 'wetdust', 'wetno3', 'wetoa', 'wetso4', 'wetss']

Pyaerocom ReadGridded
---------------------
Data ID: TM5_AP3-INSITU-TIER3
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III/TM5_AP3-INSITU-TIER3/renamed
Available experiments: ['AP3']
Available years: [2010]
Available frequencies ['hourly']
Available variables: ['abs440dryaer', 'abs550aer', 'abs550dryaer', 'abs550drylt1aer', 'abs550rh40aer', 'abs550rh55aer', 'abs550rh65aer', 'abs550rh75aer', 'abs550rh85aer', 'abs870dryaer', 'airmass', 'asydryaer', 'dh', 'ec440dryaer', 'ec550aer', 'ec550aerh2o', 'ec550bc', 'ec550dryaer', 'ec550drylt1aer', 'ec550dust', 'ec550no3', 'ec550oa', 'ec550rh40aer', 'ec550rh55aer', 'ec550rh65aer', 'ec550rh75aer', 'ec550rh85aer', 'ec550so4', 'ec550ss', 'ec870dryaer', 'hus', 'mmrbc', 'mmrdust', 'mmrmsa', 'mmrnh4', 'mmrno3', 'mmroa', 'mmrso4', 'mmrss', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'plev', 'pr', 'ta']

Pyaerocom ReadGridded
---------------------
Data ID: TM5-met2010_AP3-CTRL2019
Data directory: /lustre/storeB/project/aerocom/aerocom-users-database/AEROCOM-PHASE-III-2019/TM5-met2010_AP3-CTRL2019/renamed
Available experiments: ['AP3']
Available years: [1850, 2010]
Available frequencies ['monthly' 'daily']
Available variables: ['abs350aer', 'abs440aer', 'abs440dryaer', 'abs550aer', 'abs550dryaer', 'abs550drylt1aer', 'abs870aer', 'abs870dryaer', 'airmass', 'asyaer', 'asydryaer', 'depbc', 'depdms', 'depdust', 'dephno3', 'depmsa', 'depn', 'depnh3', 'depnh4', 'depnhx', 'depno2', 'depno3', 'depnoy', 'depo3', 'depoa', 'deps', 'depso2', 'depso4', 'depsoa', 'depss', 'dh', 'drybc', 'drydms', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'dryno3', 'drynoy', 'dryo3', 'dryoa', 'dryso2', 'dryso4', 'drysoa', 'dryss', 'ec440dryaer', 'ec550aer', 'ec550dryaer', 'ec550drylt1aer', 'ec870dryaer', 'emibc', 'emico', 'emidms', 'emidust', 'emiisop', 'emin', 'eminh3', 'eminox', 'emis', 'emiso2', 'emiso4', 'emiss', 'emiterp', 'emivoc', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadsoa', 'loadss', 'od350aer', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550lt1ss', 'od550no3', 'od550oa', 'od550so4', 'od550soa', 'od550ss', 'od870aer', 'pr', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcnh4', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcsoa', 'sconcss', 'ta', 'temp', 'vmrch4', 'vmrco', 'vmrno', 'vmrno2', 'vmro3', 'vmroh', 'wetbc', 'wetdms', 'wetdust', 'wethno3', 'wetmsa', 'wetnh3', 'wetnh4', 'wetno3', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetsoa', 'wetss', 'ang4487aer', 'angabs4487aer', 'od550gt1aer', 'vmrox', 'fmf550aer', 'concNnh4']
['TM5JRCCY2IPCCV1_SR6SA', 'TM5-JRC-cy2-ipcc-v1_SR1', 'TM5JRCCY2IPCCV1_SR6EU', 'TM5JRCCY2IPCCV1_SR1', 'TM5JRCCY2IPCCV1_SR6NA', 'TM5JRCCY2IPCCV1_SR6EA', 'TM5_B', 'TM5-V3.A2.PRE', 'TM5-V3.A2.HCA-0', 'TM5-V3.A2.CTRL', 'TM5-V3.A2.HCA-IPCC', 'TM5_AP3-CTRL2016', 'TM5_AP3-INSITU', 'TM5_AP3-CTRL2015', 'TM5_AP3-INSITU-TIER3', 'TM5-met2010_AP3-CTRL2019']

You should now see the downloaded data! Nice! This worked, and there is even a lot of additional information, that comes in handy. The latter is because the underlying NetCDF files in the data directory are stored using AeroCom file naming conventions. Each dataset has itโ€™s own ID (usually the directory name and can be accessed via this ID). For this example TM5 dataset the ID is TM5-met2010_CTRL-TEST as can be seen in the output from the browsing method.

pyaerocom makes extensive use og these conventions, which makes it easy to streamline analyses of many different models and observation records. However, as we shall see below, the latter are often formatted in many different ways, as observations from many different databases are used.

[19]:

reader = pya.io.ReadGridded('TM5-met2010_CTRL-TEST')
reader

[19]:


Pyaerocom ReadGridded
---------------------
Data ID: TM5-met2010_CTRL-TEST
Data directory: /home/lewisb/MyPyaerocom/testdata-minimal/modeldata/TM5-met2010_CTRL-TEST/renamed
Available experiments: ['AP3']
Available years: [2010, 9999]
Available frequencies ['daily' 'monthly']
Available variables: ['abs550aer', 'od550aer']

Tiny detour: AeroCom file naming conventions๏ƒ

Letโ€™s have a brief look at such a filename (taking the first file in the data directory):

[20]:

first_file = reader.files[0]
os.path.basename(first_file)

[20]:

'aerocom3_TM5-met2010_AP3-CTRL2019_abs550aer_Column_2010_daily.nc'

The template is:

aerocom3_<ModelName>-<MeteoConfigSpecifier>_<ExperimentName>-<PerturbationName>_<VariableName>_<VerticalCoordinateType>_<Period>_<Frequency>.nc

So the above filename uses TM5 model, 2010 meteorology (met2010), and this version is for AeroCom Phase III (AP3) experiment, particularly for the 2019 Control (CTRL2019) perturbation. Variable is abs550aer (which is the aerosol absorption optical depth, or AAOD), which is representative for a whole atmospheric Column, the simulated year is 2010 (here it is the same as meteorology, but this must not always be the case) and the temporal resolution is daily.

If you want to learn more about AeroCom conventions and ongoing experiments, see here.

The metadata that is extracted from the filenames is accessible via:

[21]:

reader.file_info

[21]:
var_name year ts_type vert_code data_id name meteo experiment perturbation is_at_stations 3D filename
3 abs550aer 2010 daily Column TM5-met2010_CTRL-TEST TM5 met2010 AP3 CTRL2019 False False aerocom3_TM5-met2010_AP3-CTRL2019_abs550aer_Co...
1 abs550aer 2010 monthly Column TM5-met2010_CTRL-TEST TM5 met2010 AP3 CTRL2019 False False aerocom3_TM5-met2010_AP3-CTRL2019_abs550aer_Co...
0 abs550aer 9999 daily Column TM5-met2010_CTRL-TEST TM5 met2010 AP3 CTRL2019 False False aerocom3_TM5-met2010_AP3-CTRL2019_abs550aer_Co...
4 od550aer 2010 daily Column TM5-met2010_CTRL-TEST TM5 met2010 AP3 CTRL2019 False False aerocom3_TM5-met2010_AP3-CTRL2019_od550aer_Col...
2 od550aer 2010 monthly Column TM5-met2010_CTRL-TEST TM5 AP3 CTRL2016 False False aerocom3_TM5_AP3-CTRL2016_od550aer_Column_2010...

How do I know what the variable names mean?๏ƒ

You can check all variables via pyaerocom.const.VARS, which is a dictionary-like object that allows access to variables and in most cases, provides relevant additional information such as the CF standard_name. For instance, for the above abs550aer:

[22]:

var = pya.const.VARS['abs550aer']
var

[22]:

abs550aer
standard_name: atmosphere_absorption_optical_thickness_due_to_ambient_aerosol_particles; Unit: 1

[23]:

var.long_name

[23]:

'Absorption aerosol optical depth (AAOD) at 550nm'

Or the extinction (scattering + absorption) aerosol optical depth (AOD), called od550aer:

[24]:

var = pya.const.VARS['od550aer']
var

[24]:

od550aer
standard_name: atmosphere_optical_thickness_due_to_ambient_aerosol_particles; Unit: 1

Reading of model data using ReadGridded class๏ƒ

The above instantiated ReadGridded interface relies on and makes use of these conventions. This class is also the standard interface to read the modeldata into instances of the pyaerocom.GriddedData

[25]:

aaod_tm5 = reader.read_var('abs550aer', start=2010, ts_type='monthly')
aaod_tm5

[25]:

pyaerocom.GriddedData: (abs550aer, TM5-met2010_CTRL-TEST)
<iris 'Cube' of atmosphere_absorption_optical_thickness_due_to_ambient_aerosol / (1) (time: 12; latitude: 90; longitude: 120)>

Under the hood, the GriddedData object is an iris.Cube, and it is single variable, that is, it does not support reading of multiple variable fields (e.g. AOD and AAOD sharing the same lat, lon and time dimensions).

The GriddedData object is introduced in more detail in other tutorials, but what is a tutorial without a nice, self-explanatory plot anyways?

[26]:

aaod_tm5.sel(latitude=(-30, 30), longitude=(-150, 150)).quickplot_map('06/2010');
../_images/pyaerocom-tutorials_getting_started_setup_39_0.png

Registering and reading of ungridded observational data๏ƒ

Hopefully comming soonโ€ฆ

Until then, checkout the section on ungridded observations in the following tutorial getting_started_analysis.