Newer
Older
Configuration file options
==========================
This section contains the list and explanation about all the options that are available on the configuration file. Use
it as a reference while preparing your configuration file. Each subsection will refer to the matching section from the
config file. Those subsections explanation may be divided itself for the shake of clarity but this further divisions
have nothing to do with the config file syntax itself.
This section contains the general configuration for the diagnostics. The explanation has been divided in two subsections: the first
one will cover all the mandatory options that you must specify in every configuration, while the second will cover all
the optional configurations.
Mandatory configurations
~~~~~~~~~~~~~~~~~~~~~~~~
* SCRATCH_DIR:
Temporary folder for the calculations. Final results will never be stored here.
* DATA_DIR:
':' separated list of folders to look for data in. It will look for file in the path $DATA_FOLDER/$EXPID and
$DATA_FOLDER/$DATA_TYPE/$MODEL/$EXPID
Folder containing mask and mesh files for the dataset.
* FREQUENCY:
Default data frequency to be used by the diagnostics. Some diagnostics can override this configuration or even
ignore it completely.
* DIAGS:
List of diagnostic to run, in the order you want them to run
Optional configurations
~~~~~~~~~~~~~~~~~~~~~~~
* SCRATCH_MASKS
Common scratch folder for the ocean masks. This is useful to avoid replicating them for each run at the fat nodes.
By default is '/scratch/Earth/ocean_masks'
* RESTORE_MESHES
By default, Earth Diagnostics only copies the mask files if they are not present in the scratch folder. If this
option is set to true, Earth Diagnostics will copy them regardless of existence. Default is False.
* DATA_ADAPTOR
This is used to choose the mechanism for storing and retrieving data. Options are CMOR (for our own experiments) or
THREDDS (for anything else). Default value is CMOR
* DATA_TYPE
Type of the dataset to use. It can be exp, obs or recon. Default is exp.
* DATA_CONVENTION
Convention to use for file paths and names and variable naming among other things. Can be SPECS, PRIMAVERA or CMIP6.
Default is SPECS.
* CDFTOOLS_PATH
Path to the folder containing CDFTOOLS executables. By default is empty, so CDFTOOLS binaries must be added to the
system path.
* MAX_CORES
Maximum number of cores to use. By default the diagnostics will use all cores available to them. It is not
necessary when launching through a scheduler, as Earthdiagnostics can detect how many cores the scheduler has
allocated to it.
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
EXPERIMENT
----------
This sections contains options related to the experiment's definition or configuration.
* MODEL
Name of the model used for the experiment.
* MODEL_VERSION
Model version. Used to get the correct mask and mesh files
* ATMOS_TIMESTEP
Time between outputs from the atmosphere. This is not the model simulation timestep!
* OCEAN_TIMESTEP
Time between outputs from the ocean. This is not the model simulation timestep!
* ATMOS_GRID
Atmospheric grid definition. Will be used as a default target for interpolation diagnostics.
* INSTITUTE
Institute that made the experiment, observation or reconstruction
* EXPID
Unique identifier for the experiment
* NAME
Experiment's name. By default it is the EXPID.
* STARTDATES
Startdates to run as a space separated list
* MEMBER
Members to run as a space separated list. You can just provide the number or also add the prefix
* MEMBER_DIGITS
Number of minimum digits to compose the member name. By default it is 1. For example, for member 1 member name
will be fc1 if MEMBER_DIGITS is 1 or fc01 if MEMBER_DIGITS is 2
* MEMBER_PREFIX
Prefix to use for the member names. By default is 'fc'
* MEMBER_COUNT_START
Number corresponding to the first member. For example, if your first member is 'fc1', it should be 1.
If it is 'fc0', it should be 0. By default is 0
* CHUNK_SIZE
Length of the chunks in months
* CHUNKS
Number of chunks to run
* CALENDAR
Calendar to use for date calculation. All calendars supported by Autosubmit are available. Default is 'standard'
CMOR
----
In this section, you can control how will work the cmorization process. All options belonging to this section are optional.
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
Cmorization options
~~~~~~~~~~~~~~~~~~~
This options control when and which varibales will be cmorized.
* FORCE
If True, launches the cmorization, regardless of existence of the extracted files or the package containing the
online-cmorized ones. If False, only the non-present chunks will be cmorized. Default value is False
* FORCE_UNTAR
Unpacks the online-cmorized files regardless of exstience of extracted files. If 'FORCE is True, this parameter has
no effect. If False, only the non-present chunks will be unpacked. Default value is False.
* FILTER_FILES
Only cmorize original files containing any of the given strings. This is a space separated list. Default is the
empty string.
* OCEAN_FILES
Boolean flag to activate or no NEMO files cmorization. Default is True.
* ATMOSPHERE_FILES
Boolean flag to activate or no IFS files cmorization. Default is True.
* USE_GRIB
Boolean flag to activate or no GRIB files cmorization for the atmosphere. If activated and no GRIB files are present,
it will cmorize using the MMA files instead (as if it was set to False). Default is True.
* CHUNKS
Space separated list of chunks to be cmorized. If not provided, all chunks are cmorized
* VARIABLE_LIST
Space separated list of variables to cmorize. Variables must be specified as domain:var_name. If no one is specified,
all the variables will be cmorized
Grib variables extraction
*************************
These three options ares used to configure the variables to be CMORized from the grib atmospheric files.
They must be specified using the IFS code in a list separated by comma.
You can also specify the levels to extract using one of the the following syntaxes:
* VARIABLE_CODE
* VARIABLE_CODE:LEVEL,
* VARIABLE_CODE:LEVEL_1-LEVEL_2-...-LEVEL_N
* VARIABLE_CODE:MIN_LEVEL:MAX_LEVEL:STEP
Some examples to clarify it further:
* Variable with code 129 at level 30000: 129:30000
* Variable with code 129 at levels 30000, 40000 and 60000: 129:30000-40000-60000
* Variable with code 129 at levels between 30000 and 600000 with 10000 intervals:
129:30000:60000:10000 equivalent to 129:30000-40000-50000-60000
* ATMOS_HOURLY_VARS
Configuration of variables to be extracted in an hourly basis
* ATMOS_DAILY_VARS
Configuration of variables to be extracted in a daily basis
* ATMOS_MONTHLY_VARS
Configuration of variables to be extracted in a monthly basis
Metadata options
~~~~~~~~~~~~~~~~
All the options in this subsection will serve just to add the given values to the homonymous attributes in the
cmorized files.
* ASSOCIATED_EXPERIMENT
Default value is 'to be filled'
* ASSOCIATED_MODEL
Default value is 'to be filled'
* INITIALIZATION_DESCRIPTION
Default value is 'to be filled'
* INITIALIZATION_METHOD
Default value is '1'
* PHYSICS_DESCRIPTION
Default value is 'to be filled'
* PHYSICS_VERSION
Default value is '1'
* SOURCE
Default value is 'to be filled'
THREDDS
-------
For now, there is only one option for the THREDDS server configuration.
* SERVER_URL
THREDDS server URL
ALIAS
-----
This config file section is different from all the others because it does not contain a set of configurations. Instead,
in this section the user can define a set of aliases to be able to launch its most used configurations with ease. To do
this, the user must add an option with named after the desired alias and assign to it the configuration or configurations to launch
when this ALIAS is invoked. See the next example:
.. code-block:: ini
ALIAS_NAME = diag,opt1,opt2 diag,opt1new,opt2
In this case, the user has defined a new alias 'ALIAS' that can be used launch two times the diagnostic 'diag',
the first with the options 'opt1' and 'opt2' and the second replacing 'opt1' with 'opt1new'.
In this example, configuring the DIAGS as
.. code-block:: ini
DIAGS = ALIAS_NAME
will be identical to
.. code-block:: ini
DIAGS = diag,opt1,opt2 diag,opt1new,opt2