User Tools

Site Tools

Trace:
tools:eionet-utdretriever

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
tools:eionet-utdretriever [2016/01/27 11:35]
jcuadrad Added new mode of operation (download_no_filters) 		tools:eionet-utdretriever [2016/01/27 15:00]
jcuadrad
Line 82: Line 82:
   * download: Normal operation, automated data download data uses the UpdatedSince and CreatedSince filters. In this mode, the retriever will keep track of the dates of last successful download for each pollutant and country, storing it in the DOWNLOAD_DATE table in the database.   * download: Normal operation, automated data download data uses the UpdatedSince and CreatedSince filters. In this mode, the retriever will keep track of the dates of last successful download for each pollutant and country, storing it in the DOWNLOAD_DATE table in the database.
   * download_no_filters: Manual operation, intended to troubleshoot missing data in the DB. Since the filters are not used in this mode all observations from the time window specified will be downloaded (be aware of reaching the 50k observations limit has set in place). The retriever will not keep track of the last succesful downloads in this mode.   * download_no_filters: Manual operation, intended to troubleshoot missing data in the DB. Since the filters are not used in this mode all observations from the time window specified will be downloaded (be aware of reaching the 50k observations limit has set in place). The retriever will not keep track of the last succesful downloads in this mode.
 +  * download_sliding_no_filters: Same as "download_no_filters" mode but it can be used with relative dates (to today). Please see example below.
  
 In the "download_no_filters" mode the usage is as follows: In the "download_no_filters" mode the usage is as follows:
 <code>python3 EIONETretriever.py download_no_filters GER --fromDate 2016-01-01 --toDate 2016-01-03 > logs/GER/GER-20160101-20160102.log</code> <code>python3 EIONETretriever.py download_no_filters GER --fromDate 2016-01-01 --toDate 2016-01-03 > logs/GER/GER-20160101-20160102.log</code>
 In this example, all observations of the countries (de,at,pl) and pollutants defined in the "GER.conf" file from 01/jan/2016 to 03/jan/2016 (not included) will be downloaded and stored in the database. \\ In this example, all observations of the countries (de,at,pl) and pollutants defined in the "GER.conf" file from 01/jan/2016 to 03/jan/2016 (not included) will be downloaded and stored in the database. \\
 +
 +Example for "download_sliding_no_filters": \\
 +If today is 2016-01-27 and we want to download the observations of days 2016-01-12 and 2016-01-13 we can get them in two ways:
 +<code>python3 EIONETretriever.py download_no_filters ES --fromDate 2016-01-12 --toDate 2016-01-14 > logs/ES/ES-20160112-20160113.log
 +python3 EIONETretriever.py download_sliding_no_filters ES --fromDaysAgo 15 --toDaysAgo 13 > logs/ES/ES-20160112-20160113.log</code>
 +Please note that in this mode, if the option --toDaysAgo is not provided the download will be until to date (now). \\
 +
 As usual, the --help (or -h) option will also display the command-line manual/help. As usual, the --help (or -h) option will also display the command-line manual/help.
 Please note that this article always refers to the 'download' command if not otherwise explicitly stated. Please note that this article always refers to the 'download' command if not otherwise explicitly stated.
tools/eionet-utdretriever.txt ยท Last modified: 2016/12/29 09:09 by kserrade

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki