[profiler] Perform simple tests and plot the results with different configurations
cc'ing @pgoitia
This issue is a follow-up to #1051 (closed) and #1108 (closed).
Some suggestions of test scenarios:
- A workflow configuration similar to what Dani used to profile memory issues
- A workflow configuration with no placeholders, compared to a workflow configuration with lots of placeholders
- A workflow configuration with minimal, compared to a workflow configuration with old-style configuration
- A workflow configuration with minimal, compared to a workflow configuration with minimal and many CUSTOM_CONFIG's, PRE's, POST's
- A workflow configuration with a single minimal file, that contains a lot of extra configuration entries
- A workflow configuration with a collection of many (100's, 1000's) of small YAML files.
- A workflow configuration with a collection of many (100's, 1000's) of large YAML files.
- A workflow configuration with minimal, that includes another YAML file
- A workflow configuration with wrappers 9.1. Horizontal 9.2. Vertical 9.3. Hybrid ...
We can create a directory in the code, maybe under test (e.g. test/performance/profiler-tests).
Each test could be a Jupyter notebook (bonus points if using Docker). It must contain a short paragraph containing the analysis of the results, followed by a simple 2-bar chart (matplotlib/notebook?) showing memory consumption side-by-side. We can also filter the top-N entries of the cProfile output (probably N=5 is good enough?).
The tests can be performed locally or on our VM's or MN4. I think it won't make much difference running on MN4, since the Python version and libraries can be replicated locally. IMHO, the only advantage of using MN4 would be if we were measuring I/O.