Differences

This shows you the differences between two versions of the page.

--- reproducibility [2015/12/11 12:10]
fmassonn [9 December 2015]
+++ reproducibility [2016/01/14 19:54]
xyepes [17 December 2015]
@@ Line 258: / Line 258: @@
 The discussions were quite rich, and here is the summary in a few bullet points
-  * We have to be extremely **careful** when saying things like "EC-Earth is not reproducible". First because "reproducibility" is a loosely defined concept: bit-for-bit reproducibility is different from climate-for-climate reproducibility. Second because we //users// might offend //developers// who strive to make their models reproducible, and this could be seen as a lack of respect.
+  * We have to be extremely **careful** when saying things like "EC-Earth is not reproducible". First because "reproducibility" is a loosely defined concept: bit-for-bit reproducibility is different from climate-for-climate reproducibility. Defining the latter (i.e., are two ensembles statistically indistinguishable from each other) is particularly challenging, both regarding what protocol to use and the statistical test to apply. The other reason why we need to be careful is because we //users// might offend //developers// who strive to make their models reproducible, and this could be seen as a lack of respect.
-  * SMHI is mostly interested in understanding what are the configurations under which EC-Earth is reproducible, while the initial question we (at IC3) asked back then was: can we run EC-Earth on different platforms if we follow our common standards.
+  * SMHI is mostly interested in understanding what are the configurations under which EC-Earth is reproducible, while the initial question we (at IC3 and now BSC) ask is: can we run EC-Earth on different platforms if we follow our common standards.
+  * Assessing reproducibility of a whole system is different from assessing reproducibility of one particular variable (e.g., Antarctic sea ice extent in winter). A good point of the Barker et al. paper referenced above is that their test is multivariate, since the set of 120 variables is first EOFed. By making the study in the Principal Component Analysis (statistical) space rather than physical space, they reach probably stronger conclusions than if they had looked at all (dependent) variables separately, as we do in our case.
+The topic is becoming extremely complex, far-reaching and our team looking into the topic is growing every month. On the other hand it has been a long-standing issue (almost one year now) and we need to have insights for the next EC-Earth meeting and the upcoming CMIP6. Here is a suggestion as how to continue the work: this should be split in two tasks
+  * **Developer aspect** - Xavi Yepes is now looking in the bit-for-bit reproducibility issue with EC-Earth 3.2. and for short (3-month) runs. SMHI (Uwe, Klaus) and KNMI (Philippe Le Sager) are aware of this. He is making several tests:
+         - Changing the number of processors in NEMO, IFS, both.
+         - Setting optimization to -O2 or -O3
+         - Setting the -fp-model to precise, strict, source
+  * **User aspect** - Asif, François will continue adopting the "user" point of view, extend i06c under the same conditions as before, in order to reach same conclusions as before but without the massive drift that we had. When insights from the "developer", team will be available, other tests will be performed to see if we can achieve reproducibility or not.
+===== 15 December 2015 =====
+The **User aspect** experiments are launched. Ithaca's **i077** is now under way.
+===== 17 December 2015 =====
+François and Xavier agreed that it is necessary to perform several executions changing technical aspects. Ideally, the following aspects should be all evaluated, but it is not feasible to handle it, because combinations grow exponentially. So, the parameters to try are:
+  * Compulsory:
+    * Code optimization: -O2 and -O3
+    * Regarding floating-point calculations: -fp-model [precise | strict]
+    * Usage of -xHost flag (best instructions according to host machine)
+    * Two processor combinations:
+      * IFS 320 and NEMO 288
+      * IFS 128 and NEMO 64
+  * Optional:
+    * Without -xHost
+    * Without -fp-model clause
+    * Try -fp-model source
+    * Explore more processor combinations
+So, we should have 4 compulsory compilations:
+  * -O2 -fp-model precise -xHost
+  * -O2 -fp-model strict -xHost
+  * -O3 -fp-model precise -xHost
+  * -O3 -fp-model strict -xHost
+And consequently, 8 compulsory outputs.
+Additional considerations:
+  * Use last EC-Earth 3.2beta release
+  * Enable key_mpp_rep
+  * 1 month, writing every day
+  * Use optimization to avoid mpi_allgather use at the northfold

User Tools

Site Tools

Differences

Page Tools