This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
working_groups:ukurbangroup [2022/08/05 12:26] mhajji old revision restored (2022/08/05 13:57) |
working_groups:ukurbangroup [2022/09/19 09:46] acriado |
||
---|---|---|---|
Line 4: | Line 4: | ||
Universal Kriging is a common geostatistic technique used for spatial interpolation, | Universal Kriging is a common geostatistic technique used for spatial interpolation, | ||
- | spatial structure of the data—. In our case, we have applied this methodology as a post-process of the CALIOPE-Urban dispersion model, developed by the Earth Science Department of the Barcelona Supercomputing Center (BSC). To implement it, we have used the hourly observational NO2 data coming | + | spatial structure of the data—. In our case, we have applied this methodology as a post-process of the CALIOPE-Urban dispersion model, developed by the Earth Science Department of the Barcelona Supercomputing Center (BSC). To implement it, we have used the hourly observational NO2 data from 12 monitoring stations as the principal variable and the CALIOPE-Urban hourly NO2 output as the covariate. In addition, we have studied the added value to incorporate as the second covariate our |
time-invariant microscale-Land Use Regression (LUR) model, developed by using two different NO2 passive dosimeters campaigns and 8 predictors (urban geometric variables, simulated vehicular traffic densities, annually-averaged data bi-linearly interpolated from the regional CALIOPE system and the annually-averaged NO2 output of CALIOPE-Urban) through a machine learning approach. Our implementation is a data-fusion procedure used as a spatial NO2 bias correction in an urban area, the city of Barcelona. Moreover, this correction can be applied directly to the daily maximum NO2 concentrations, | time-invariant microscale-Land Use Regression (LUR) model, developed by using two different NO2 passive dosimeters campaigns and 8 predictors (urban geometric variables, simulated vehicular traffic densities, annually-averaged data bi-linearly interpolated from the regional CALIOPE system and the annually-averaged NO2 output of CALIOPE-Urban) through a machine learning approach. Our implementation is a data-fusion procedure used as a spatial NO2 bias correction in an urban area, the city of Barcelona. Moreover, this correction can be applied directly to the daily maximum NO2 concentrations, | ||
- | the implementation of this methodology is under revision. | + | implementing |
{{ : | {{ : | ||
Line 16: | Line 16: | ||
__Meriem Hajji__, meriem.hajji@bsc.es | __Meriem Hajji__, meriem.hajji@bsc.es | ||
- | ===== First Steps ===== | ||
- | To follow a basic tutorial using this methodology, | + | ===== Visualization ===== |
- | === Using the GitLab repository and copying all the needed archives === | + | At this point, it is recommended to follow |
- | - Copy all the functions and archives needed to implement the procedure in your own directory. To do that, open a terminal and copy the following command: | + | ===== First Steps : understanding |
- | <code bash> git clone https:// | + | |
- | After doing that, a folder called by default // | + | |
+ | Please follow the next steps to follow a basic tutorial using this methodology. The procedure is implemented using the R software. | ||
+ | === Using the GitLab repository and copying all the needed archives === | ||
- | In the folder | + | - Copy all the functions and archives needed to implement the procedure in your directory. To do that, open a terminal and copy the following command: |
+ | <code bash> git clone https://earth.bsc.es/gitlab/es/ | ||
+ | After that, a folder called by default // | ||
- | + | A list of different archives will appear in the folder // | |
- | + | ||
- | At this point, it is recommended to follow | + | |
=== The configuration file === | === The configuration file === | ||
- | The configuration file is an archive used as a setup structure, | + | The configuration file is an archive used as a setup structure, |
* Through the Rstudio visualization: | * Through the Rstudio visualization: | ||
{{ : | {{ : | ||
Line 50: | Line 49: | ||
- | * **// | + | * **// |
- | * **// | + | * **// |
- | * **// | + | * **// |
* **// | * **// | ||
Line 62: | Line 61: | ||
* //UK//: Universal Kriging hourly correction (the default). | * //UK//: Universal Kriging hourly correction (the default). | ||
* //cross//: If we want to apply the Leave-One-Out Cross-Validation, | * //cross//: If we want to apply the Leave-One-Out Cross-Validation, | ||
- | * //UK_max// and // | + | * //UK_max// and // |
- | * // | + | * // |
* // | * // | ||
Line 69: | Line 68: | ||
* **// | * **// | ||
* **// | * **// | ||
- | * **// | + | * **// |
As a resume, notice that the user only has to do the following about the configuration file: | As a resume, notice that the user only has to do the following about the configuration file: | ||
* Choosing you own **// | * Choosing you own **// | ||
* Choosing one of the **// | * Choosing one of the **// | ||
- | * Choosing the Universal Kriging mode in terms of the covariates, the application, | + | * Choosing the Universal Kriging mode in terms of the covariates, the application, |
* (The rest of the configuration should not be changed, only fill it as is explained). | * (The rest of the configuration should not be changed, only fill it as is explained). | ||
=== The structure of the folders === | === The structure of the folders === | ||
- | Before applying the methodology and obtaining the results, is important | + | Before applying the methodology and obtaining the results, |
This is an example of the structure using the 2019 dataset : | This is an example of the structure using the 2019 dataset : | ||
{{ : | {{ : | ||
- | Remember that the parallelization is carried out in terms of the day. We are applying the methodology on a mesh composed approximately | + | Remember that the parallelization is carried out in terms of the day. We are applying the methodology on a mesh of approximately |
=== The main script and its explanation | === The main script and its explanation | ||
- | The // | + | The // |
* The first section, **//Config file//**, is about setting up the procedure by reading the configuration file. | * The first section, **//Config file//**, is about setting up the procedure by reading the configuration file. | ||
* The section **//initial setting//** takes into account: | * The section **//initial setting//** takes into account: | ||
- | - The principal | + | - The top libraries that will be used and the coordinates reference. |
- The pollutant, in this case we have to type //NO2// and //no2// for the model and observations configurations respectively. | - The pollutant, in this case we have to type //NO2// and //no2// for the model and observations configurations respectively. | ||
- The variogram' | - The variogram' | ||
Line 98: | Line 96: | ||
* In the section **//initial setting: directories// | * In the section **//initial setting: directories// | ||
* In the section **//initial setting: variables// | * In the section **//initial setting: variables// | ||
- | - All regarding dates is being set up. | + | - All regarding dates are being set up. |
- | - The mesh where the bias correction | + | - The mesh where the bias correction |
* In the section **// | * In the section **// | ||
* In the section **//caliope evaluation, mean and max//**, different scripts are used to prepare the files regarding the model (CALIOPE-Urban) output at the monitoring stations (caliope evaluation), | * In the section **//caliope evaluation, mean and max//**, different scripts are used to prepare the files regarding the model (CALIOPE-Urban) output at the monitoring stations (caliope evaluation), | ||
* In the section **// | * In the section **// | ||
- | * The section // | + | * The section // |
* In the section // | * In the section // | ||
* In the section **// | * In the section **// | ||
* In the section **// | * In the section **// | ||
- | Notice that the majority of the __// | + | Notice that the majority of the __// |
- | === Submitting jobs === | + | === First Steps : Submitting jobs === |
- | *It is recommended to first take a look at the guidelines | + | *It is recommended |
- | To apply the methodology, | + | To apply the methodology, |
- | * //#SBATCH --time=48: | + | * //#SBATCH --time=48: |
* //#SBATCH --time=01: | * //#SBATCH --time=01: | ||
Line 145: | Line 143: | ||
==== Universal Kriging using CALIOPE-Urban as the unique covariate and the whole 2019 data, using all the possible applications ==== | ==== Universal Kriging using CALIOPE-Urban as the unique covariate and the whole 2019 data, using all the possible applications ==== | ||
- | - Preparing the configuration file: in this case the same as the example above. | + | - Preparing the configuration file: This is the same as the example above. |
- Enter the machine: | - Enter the machine: | ||
<code bash> | <code bash> | ||
Line 152: | Line 150: | ||
| | | | | | ||
| .-.--_ | | .-.--_ | ||
- | | ,´,´.´ `. | | + | | ,','.' |
| | | | BSC | | | | | | | BSC | | | ||
- | | `.`.`. _ .´ | | + | | `.`.`. _ .' |
| `·`·· | | `·`·· | ||
| | | | | | ||
Line 187: | Line 185: | ||
* //#SBATCH --time// : the computational time required | * //#SBATCH --time// : the computational time required | ||
* //#SBATCH -n// : the machine' | * //#SBATCH -n// : the machine' | ||
- | * //#SBATCH --constraint// | + | * //#SBATCH --constraint// |
As it would be the first submitted job, we use the maximum computational time (48h) and in this case, we choose to use 50 cores. The queue has to be //bsc_es// in this case. | As it would be the first submitted job, we use the maximum computational time (48h) and in this case, we choose to use 50 cores. The queue has to be //bsc_es// in this case. | ||
Line 221: | Line 219: | ||
* //USER//: the user's number. | * //USER//: the user's number. | ||
* //ST//: the status of the job, first if it is pending (PD) or running (R). Other options are completed (CD), completing (CG), failed (F), preempted (PR), suspended (S) or stopped (ST). All of this can be seen in the machine' | * //ST//: the status of the job, first if it is pending (PD) or running (R). Other options are completed (CD), completing (CG), failed (F), preempted (PR), suspended (S) or stopped (ST). All of this can be seen in the machine' | ||
- | * //TIME//: the time that has passed since the job is running. | + | * //TIME//: the time that has passed since the job started. |
* //NODES//: the machine' | * //NODES//: the machine' | ||
* // | * // | ||
Line 228: | Line 226: | ||
{{ : | {{ : | ||
- | 6. Waiting until the job is finished. Notice that when a job is submitted, two files are created: the output and the error ones (the user has defined their names in the job directories). In the output file, the user can visualize some indications that appear while running the job. In the error one, the user can see secondary errors that maybe were not enough to cancel the job's running | + | 6. Waiting until the job is finished. Notice that when a job is submitted, two files are created: the output and the error ones (the user has defined their names in the job directories). In the output file, the user can visualize some indications that appear while running the job. In the error one, the user can see secondary errors that may not be enough to cancel the job's running |
7. The job is completed. A folder named by **//2019/ //** will be created at the **// | 7. The job is completed. A folder named by **//2019/ //** will be created at the **// | ||
Line 258: | Line 256: | ||
ssh bscXXXXX@nord4.bsc.es | ssh bscXXXXX@nord4.bsc.es | ||
</ | </ | ||
- | 3. Preparing the job, in this case we are going to reduce the number of cores and computational time, so change the queue too, and not require high memory. | + | 3. Preparing the job, in this case we will reduce the number of cores and computational time, so change the queue too, and not require high memory. |
<code bash> | <code bash> |