Improved Autosubmit archive
Hi,
As you know, there is a problem with experiment's data size in /esarchive that prevents us from moving Autosubmit to another better storage. In the past, we tried to do massive archivings running autosubmit archive
in batches for several older experiments. However, many of them had non-committed changes in their project directories, preventing autosubmit
to archive them.
Prohibiting runs of non-versioned code (#342) could help the cleaning process, but would also make the development/debugging process slower, so I think it is a strict solution that we should only adopt after careful thinking.
Lately, I thought about another possible solution that would allow us to archive
most of the experiments without implementing those restrictions. I thought about adding a feature in Autosubmit based on creating patches for non-committed changes. Then we would only need to save the compressed patch, which would be applied when unarchiving the experiment.
In order to create/apply patches, we could use git diff
/ git apply
:
But not all of our workflows are based on git, so maybe it's better to use just diff
/ patch
:
https://michlstechblog.info/blog/creating-and-applying-a-patch-with-diff-and-patch/
If we can develop such a solution, I think that in the future we will be able to periodically archive stopped experiments.