Home.md

+# Introduction
+
+Nemo modelfactors is a wrapper made to simplify the process of obtaining basic performance metrics from NEMO using the BSCTOOLS set.
+
+The script does all the required modifications to the NEMO configuration and arch files in order to be able to obtain traces from a NEMO execution with extrae, afterwards, the traces are processed and finally the performance metrics are generated using the basicanalysis tool.  
+
+The script was developed using NEMO 4.2 and it is not tested in other versions. For this reason is highly recommended to have installed that NEMO version.
+
+# Parameters
+
+Nemo modelfactors requires parameters in order to work properly, those are added by modifying the perf_metrics.config file.
+
+**The mandatory variables that need modifications are the following:**
+
+* Nemo_path (string) : Path to nemo installation folder containing the cfgs and arch dirs.
+* Nemo_input_data (string): Path to the input data needed to run the nemo cfg.
+* Compilation_arch (string): Architecture used (without the -arch suffix and the .fcm).
+* Jobs_scheduler (string): Available (slurm/lsf/torque).
+* Modules (string): List of modules loaded.
+
+**The full list can be found bellow:**
+
+* Output (string): Path where the Output dir, containing all the output files, will be created.
+* Nemo_path (string) : Path to nemo installation folder containing the cfgs and arch dirs.
+* Nemo_input_data (string): Path to the input data needed to run the nemo cfg.
+* Nemo_cores (array): List of nºcores used for executing Nemo, ( 4 48 ) makes the script execute and get Nemo traces with 4 and 48 cores. 2 different nºcores are needed to obtain scalability data.
+* Jobs_n_cores (integer): nºcores used for other jobs like compiling nemo.
+* Jobs_cores_per_node (integer): define the number of cores per node.
+* Jobs_scheduler (string): Available (slurm/lsf/torque).
+* Jobs_time (integer): Max duration of the job in min.
+* Jobs_queue (string): Queue used.
+* Compilation_compile (string): When false only compiles NEMO if arch file lacks the needed flags, when true always compiles NEMO.
+* Compilation_ref (string): Reference configuration.
+* Compilation_arch (string): Architecture used (without the -arch suffix and the .fcm).
+* Compilation_name (string): Name of the new configuration (Important to not be an existing one).
+* Compilation_sub (string): Add or remove sub-components.
+* Modules (string): List of modules loaded.
+
+# Output
+
+Nemo modelfactors generates a Output folder during execution.The location of this folder can be modified in perf_metrics.config, this allows having the main script in a low capacity file system while the heavy data is stored somewhere else.
+
+It contains:
+
+- **logs:** Contains all the error and output files of the jobs submitted during the script execution. If any job fails the script is stopped, this way you can troubleshoot the error by easily locating and reading the logs.
+- **Run_Nemo:** Is where the NEMO execution will be produced, contains all the input data, the executable and the output files of NEMO.
+- **Metrics:** Folder where the performance metrics data produced is stored, more information in the next section.
+
+# Data generated
+
+All the data generated can be found inside the Metrics folder.
+
+* **Overview file "overview.txt":** Contains a summary of the most important and easy to understand metrics such as Speedup, IPC and Frequency, and Global efficiency.
+* **Excel files (csv)**
+  * **efficiency_table.csv:** Contains all the metrics related with the efficiency, generates scalability metrics only if at least two traces with different number of cores are present.
+  * **modelfactors.csv:** Contains the same data as "efficiency_table.csv" but the decimal values have 6 digits of precision instead of 2.
+  * **other_metrics.csv:** Contains other valuable metrics such as the speedup, number of processes, elapsed time etc.
+  * **rawdata.csv:** Contains all the not processed data obtained directly from the trace and the hardware counters.
+* **Plots (png)**
+  * **efficiency_table-matplot.png:** Data from the efficiency_table.csv transformed to a table using matplot.
+  * **(2 traces minimum) efficiency-matplot.png:** Plot representing the evolution of efficiency with the increase of cores and compared with the ideal efficiency.
+  * **(2 traces minimum) modelfactors-comm-matplot.png:** Plot representing the evolution of communication, transfer and serialization efficiency related with the increase of cores.
+  * **(2 traces minimum) modelfactors-matplot.png:** Plot representing the evolution of all the efficiency related metrics with the increase of cores.
+  * **(2 traces minimum) modelfactors-scale-matplot.png:** Plot representing the evolution of the scalability metrics with the increase of cores.
+  * **(2 traces minimum) speedup-matplot.png:** Plot representing the evolution of the speedup with the increase of cores and compared with the ideal speedup.
+* **Traces:** All the traces of different number of cores corresponding to a single time-step of the NEMO execution.
+* **scratch_out_basicanalysis:** Folder containing stats and traces of the simulations created with DIMEMAS.
+
+  # Workflow
+
+  In this section the general workflow of the script is quickly explained in order to get an overview of what the script is doing.
+  1. Nemo is compiled with the -gprof flag (only if not done already).
+  2. A list of important functions is created thanks to the gprof data (only if not done already).
+  3. Nemo is compiled with the flags needed to obtain a trace using EXTRAE (only if not done already).
+  4. For every nºcores in the array defined in perf_metrics.config:
+     1. Nemo is executed and the trace is generated.
+     2. A single time_step is subtracted from the trace using the functions list and PARAVER.
+  5. Basic analysis tool is used on the cut traces to obtain the metrics.
\ No newline at end of file