Workflow optimizations squashed

Closes #1158

See merge request es/autosubmit!395

feedback fix

added :

added \n

Added permissionError

Traces hidden

Changed pkl open

Added replace instead of rename

changed to not create instead of monitor

Changes in save/load

added --no-requeue

fix

set again retrials to 3

Added removal of duplicated jobs

test platform job_id removal

added placeholder

reset fail_count on setstatus from failed to X and if graph is new

reset fail_count on setstatus from failed to X and if graph is new

Changed a test config , fixed issue with workflow generation

Changed a test config , fixed issue with workflow generation

check defaulted to False instead of True

Changed blank message to warning

standarized on_submission

debug line

configparser version outdated

Fix issues with totaljobs, check_script

Clean outdate code

added a missing counter for totaljobs

added a missing counter for totaljobs

import change location

simplified send_command

simplified if

remove unused array

Fix monitor not showing the correct status of the workflow when -cw is prompt

clean code

keys_to_erase

regex

changed configparser version

Fixed the "%CURRENT_*% issue

Fixed the "%CURRENT_*% issue

added job.packed = False once the job is ready or completed/failed (affects reruns)

fixed queue and partition not being updated properly

delete_edgeless_jobs changed

changed == for is to evaluate the type

removed get_all_filter_jobs

changed _create_jobs_splits

Now, if -cw is used in monitor or inspect. It won't check all the templates. Update configparser version to fix load/save

Fixed a perfomance issue
Fixed an issue with running: once adding more edges

fix run

Clean the code a bit

working now needs some cleaning

test

fix bug

fix bugs, reworked a bit

fix bug

updated test

updated test

fixed some bugs, added some docs

more fixes

test

fix pipeline

fix pipeline

math readded

datetime readded

fix issue with delay retrial

fix issue with -1

Fixes !es/autosubmit#1209

 fix grouping

test

test fix

test fix

some changes for Bruno comments

moved if inside gen

Workflow optimizations added ( mega squashed commit )

Fixes #1158

added zipp dependency ( rocrate in bscearth000)

re-added additional files

Database is locked error in historic db ( I think it is an issue in my computer as happened in master aswell)

QOL when splits is introduced with "" ( testing francesc experiment )

Ran regression test, noticed issue with experiment a005 and fixed

converse job to list

changed == for in to dont care about spaces

Fix splits when * and not * is in same line

added if not monitor

Fix changes

Fix delay

Fixed edge_info

Differences fixed

Differences fixed

comments fixed

comments

added comments

added N-1

deleted test of deleted function

deleted old code

fixed pipeline

Fixed save

Added version and hpcarch as requisites to change

Improved split_to

Improved split_to (wip)

Added "previous" filter (wip)

Added "previous" filter

fixed status

.lower() added

Add filter previous

docs

python3 or pytho2 ( fixed)

type python

updated test

changed configparserversion

better detection if data is changed

working, added the real configuration to the docs

changed configparserversion

working?

changed test

working?

issue_with_none

Added -f flag to force the recreation from 0 ... (useful mainly for test )

maybe almost working

fixed bug with chunk

wrapper fix

comments

comments

comments

comments

comments

comments

doble #

job_section

comments

docstring

added ref

todo

changed

wallclock

commented

removed funcy

Deleted funcy,

updated configar paser that has some fixes in changed files

Improved the run/monitor speed.

Fixed some default stuff

fix stats

Some memory changes introduced

added more cases

reformat

Added test_dependencies

changed the location

re-added marked_status

File parameter

reviewing

changed results

removed root = None

update_genealogy clean unused code

update_genealogy clean unused code

reviewing comments

reviewing comments

reviewing comments

tests

tes

fix pipeline

test fix

test fix

added funcy to setup.py

updated test

changed configparserversion

better detection if data is changed

working, added the real configuration to the docs

changed configparserversion

working?

changed test

working?

issue_with_none

Added -f flag to force the recreation from 0 ... (useful mainly for test )

maybe almost working

fixed bug with chunk

wrapper fix

comments

comments

comments

comments

comments

comments

doble #

job_section

comments

docstring

added ref

todo

changed

wallclock

commented

removed funcy

Deleted funcy,

updated configar paser that has some fixes in changed files

Improved the run/monitor speed.

Fixed some default stuff

fix stats

Some memory changes introduced

reviewing changes (comments)

reviewing changes (comments)

reviewing changes (comments)

reviewing changes (graph enumerate)

reviewing changes ( delete commentS)

reviewing changes ( delete valid parents)

reviewing changes

reviewing changes

reviewing changes

reviewing changes

reviewing changes

reviewing changes (numpy)

reviewing changes (numpy)

reviewing changes ( docstring)

reviewing changes ( docstring)

reviewing changes

reviewing changes

reviewing changes

reviewing changes

added more cases

reformat

Added test_dependencies

changed the location

re-added marked_status

File parameter

reviewing

changed results

removed root = None

update_genealogy clean unused code

update_genealogy clean unused code

reviewing comments

reviewing comments

reviewing comments

tests

tes

fix pipeline

test fix

test fix

added funcy to setup.py

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

Merge lastest changes
Fixed ext header to work under this version
Fixed default type

[rocrate] Add RO-Crate support to Autosubmit.

This commit includes work from several other commits, squashed.
It started around February 2023, and by July 2023 it was validated
by the RO-Crate community, thanks especially to Simone Leo.

Unit tests and documentation were added as well. It add support
to the following three RO-Crate profiles in Autosubmit:

- Process Run Crate
- Workflow Run Crate
- Workflow RO-Crate profile 1.0

This is available through the Autosubmit commands archive
and unarchive.

revise the changes

update version

bug fix an issue with additional_files and \\ variables

added retrial key

Move temp folder to the outside of for loops to reduce file creation. Rewrite the assertion part

Add dani's check so that it doesnt complain with file not found when proj type is none

add extended header and tailer documentation

test if the file does not exist, it throws an exception

test all the routes from extended tailer and header except fetching the file

change the check of hashbang to the first two characters

Handle if user sets value with empty key

Add R, Bash, and python extended scripts

Fix an issue with retrials ( present in 4.0) found while testing a full run with templates and wrapper

Added platform_name to the variables to load before the rest, ( mainly when building the dict )

Fixed -cw in create, like in inspect

Re-adapted some test-cases to match new code

workflows fixed

fixing all workflows

fixing all workflows

fixing all workflows

                    # If parent and childs has the same amount of splits \\ doesn't make sense so it is disabled

Remove cycles ( job depends on itself)

detail is now a function

Added a local test to compare workflows from 4.0 to 4.1 using -d option

fix default values

fix split

fix split

fixed parent.split == child.split when 1//2

improved test

added get_jobs_filtered test

Improved job_list test

Improved job_list test

pipeline not working

pipeline not working

removed __eq__ due being incompatible with grand part of the code, changed the test instead

added job_list generate tests
Added __eq__
fixed an issue with dependencies None

Changed DB for PKL in tests

Added more tests

Added more tests

fix wrapper dic

added run_member test

added test_build_job_with_existent_job_list_status  test

added compare_section  test

added update_parameters  test

added update_parameters  test

added update_parameters  test

added add_child test

added _repr test

Old tests working

Only 19 remains, have to doble check grouping

fix job_list

half fix job_list

half fix job_list

fix test_job.py

fix checkpoint and doc tests

Fix member_from

more changes

numpy deleted from environment.yml

pep warning fix

added test

fix doc

docs for the new autosubmit_rc env variable

docs for the new autosubmit_rc env variable

fix doc

added another suppress

added comment

changed try: except for suppress

 -

commented the debug line

Changed version

Changes to th efunction, fix a bug with the connection, added a close for ._transport of ssh

more fixes

added a debugfunction

Added a notify for push

force portalocker to <= 2.7

removed inputtimeout from requeriments

requeriments

2fa notification change

Fix applied to 2fa, local platform may were asking for a password

Fix applied to 2fa

indent in docs

dependencies

docs

docs

added method parameter

2fa: instead of 2fa

rollback few things

2fa threads

timeout

timeout

test 2fa

added docs

CHANGED input for getpass to hide typing ( it may not work)

2fa

2fa

fix additional files for ecmwf

Fixed more issues, now edgeless nodes are correctly deleted and dependencies parameter is correctly set , fixed other issues when loading previous job_list and when the node doesnt have the job

fixed few workflow inconsistencies

fixed dependency

fixed ready jobs

more fix

Working but have an issue with the initial status

added apply_filter_1_to_1

more test

test

more fixes

bsic monitor working

working on fixing merges

working on fixing merges

Pickle working,
Futher performance improves in the manage_dependencies part

working with pickle up to 1000000, afterwards it give segfualt in saving.. looking for alternatives

MUCH faster, is probabily bugged for some cases (wip)

version update

Added a delete function for nodes that are no longer part of the workflow ( with a xor)

TODO:

Delete old nodes

Reloading only the neccesary, added two methods for asconfparser

Fix reload in create

pkl changes

working faster, no memory issues but thinking more solutions

corrected prents

testing

fast test

Fixed some bugs with refactor

More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method,  code for generate jobs run very fast, inspect working has to check other commands

Reduced uneccesary operations,
Reduced memory usage

Using igraph for perform the transitive reduction

added split filter

added split filter

setstatus refactoring

rebased

pkl changes

working faster, no memory issues but thinking more solutions

corrected prents

testing

fast test

Fixed some bugs with refactor

More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method,  code for generate jobs run very fast, inspect working has to check other commands

Reduced uneccesary operations,
Reduced memory usage

Using igraph for perform the transitive reduction

added split filter

added split filter

setstatus refactoring

cherry picked fix ( changes in nodes declaration )

total_jobs fix

suppress

changed assert

added unit test

added unit test

added unit test

split the function in some smaller functions to add unit testing easier

split the function in some smaller functions to add unit testing easier

split the function in some smaller functions to add unit testing easier

fix wrapper and totaljobs

added retrial, command is failing sometimes

removed submit_cmd from update_cmds

added .get

moved - at the end

changed requeriments

now all regex includes _.-

update_version

changes

Add mailmap file

(cherry picked from commit 4f76976f)

added --exclusive

changed the bugfix

sleep rechanged

fix bug

Changed to pass ro_crate and be more robust to miss introductions

Changed to pass ro_crate and be more robust to miss introductions

Stats now counts the exact amount of processors if node is prompt

Queuing times for inner_jobs fixed

updated test

updated test

changed version

.lower() added

Add filter previous

changed name

changed name

added test ( same than 4.1)

version

Fixes #1158

changed version

Add SECURITY.MD file

Add documentation about exit status of autosubmit run

Return >0 if the autosubmit command fails

.lower() added

Add filter previous

changed name

changed name

added test ( same than 4.1)

fixed bug with chunk

version

wallclock

Fixes #1158

Fixes #1158

added zipp dependency ( rocrate in bscearth000)

changed version

Fix: This commit fixes an error message inconsistency

See merge request es/autosubmit!402

Add mailmap file

See merge request !393

Add SECURITY.md

Closes #884

See merge request !363

Return >0 if the autosubmit command fails

Closes #1038

See merge request es/autosubmit!390

Produce RO-Crate archive with Autosubmit

Closes #929

See merge request !317

This commit includes work from several other commits, squashed.
It started around February 2023, and by July 2023 it was validated
by the RO-Crate community, thanks especially to Simone Leo.

Unit tests and documentation were added as well. It add support
to the following three RO-Crate profiles in Autosubmit:

- Process Run Crate
- Workflow Run Crate
- Workflow RO-Crate profile 1.0

This is available through the Autosubmit commands archive
and unarchive.

bug fix an issue with additional_files and \\ variables

See merge request !389

Extended header and tailer for Autosubmit 4

See merge request !376

# Conflicts:
#	docs/source/userguide/configure/index.rst
#	test/unit/test_job.py

Fix typos, comments, unreachable code, and one possible bug

See merge request !386

Fix filter

Closes #1147

See merge request !385

Remove numpy, replace by math module and pure python

Closes #1142

See merge request !381

# Conflicts:
#	docs/source/installation/index.rst

Autosubmitrc custom location

See merge request es/autosubmit!384

Docs custom directives

See merge request !383