Earth Sciences

Earth Sciences Wiki

User Tools

Site Tools


data:data_repo_conventions

General considerations

  • There are 4 folders inside /esarchive/: exp (experiments), obs (observations), recon (reconstructions) and oper (operations)
  • All names must be in lowercase. The only exception is the folder names of the projects, which all must be capitalised
  • Hyphens (‘-’) can be used in the names of institutions, models, datasets and experiments to facilitate the reading. Hyphens can not be used in variable names, as they are used with other meanings (to indicate the resolution)
  • There are no underscores (‘_’) in the names of the datasets, models, etc. Only to specify the version used. To make the names more readable, you should use hyphens (‘-’)
  • Datasets without information about the version in their names refer to the first version of the dataset
  • Inside ‘constant’ folders, are stored the masks for that experiment, model or dataset
  • If ${res} is not specified, then it refers to the original/global grid. If it specified, it should be of the type ‘512×256’ for example.
  • In the case of the ${frequency}_mean data, the name of the ${var} subfolder will always contain information of the frequency at which the data was obtained ([_(f${freq}|s0-12)]), as long as that information is available.
  • The models folders inside projects folders (/esarchive/exp/${PROJECT}/${model} or esnas/exp/${PROJECT}/${exp_time_scale}) contain symbolic links pointing to the datasets stored into models folders (/esnas/exp/${model}), linked by the experiment
  • The ‘experiment time scale’ folders can contain ${frequency}_mean folders with data about all the members and models differentiated by variable
  • The original cmorfiles are stored in /esnas/exp/${model}/${expid}[_${extraid}]/cmorfiles/, and they are accessed from other paths by link
  • General scripts that operate on several subsets of a model or dataset can be placed at the top level folder inside the model/dataset, more particular scripts can be stored in the specific subfolder inside the model/dataset. A particular scripts folder after obs or recon to store global configuration files.
  • A README file with a description of the model or dataset available at the root directory of that model/dataset
  • A page on the department wiki with the inventory of the different datasets and some information about them
  • A docs folder to store any relevant document related to the dataset, model or experiment
  • Special case: for the monthly forecast and hindcast systems (CMA, JMA, ECMWF) there is an extra directory after ${var} corresponding to the date for associated forecast or hindcast
  • All data files can have associated a hash file that will contain the results of applying the hash to that file. This files should be in the same folder and the name must be .${DATA_FILE_NAME}.${HASH_ALGORITHM}.hash
data/data_repo_conventions.txt · Last modified: 2019/02/07 11:30 by fbeninca