logs rework

logs

Working

some changes and trying to use process instead of thread

Fixed a perfomance issue
Fixed an issue with running: once adding more edges

logs

fix run

Rework of Retrieval of logs parcially working

Now remote works but local not

Rework of Retrieval of logs parcially working

log_recovery rework

wip logs change

Clean the code a bit

working now needs some cleaning

test

fix bug

fix bugs, reworked a bit

fix bug

updated test

updated test

fixed some bugs, added some docs

more fixes

test

fix pipeline

fix pipeline

math readded

datetime readded

fix issue with delay retrial

fix issue with -1

Fixes !es/autosubmit#1209

 fix grouping

test

test fix

test fix

some changes for Bruno comments

moved if inside gen

Workflow optimizations added ( mega squashed commit )

Fixes #1158

added zipp dependency ( rocrate in bscearth000)

re-added additional files

Database is locked error in historic db ( I think it is an issue in my computer as happened in master aswell)

QOL when splits is introduced with "" ( testing francesc experiment )

Ran regression test, noticed issue with experiment a005 and fixed

converse job to list

changed == for in to dont care about spaces

Fix splits when * and not * is in same line

added if not monitor

Fix changes

Fix delay

Fixed edge_info

Differences fixed

Differences fixed

comments fixed

comments

added comments

added N-1

deleted test of deleted function

deleted old code

fixed pipeline

Fixed save

Added version and hpcarch as requisites to change

Improved split_to

Improved split_to (wip)

Added "previous" filter (wip)

Added "previous" filter

fixed status

.lower() added

Add filter previous

docs

python3 or pytho2 ( fixed)

type python

updated test

changed configparserversion

better detection if data is changed

working, added the real configuration to the docs

changed configparserversion

working?

changed test

working?

issue_with_none

Added -f flag to force the recreation from 0 ... (useful mainly for test )

maybe almost working

fixed bug with chunk

wrapper fix

comments

comments

comments

comments

comments

comments

doble #

job_section

comments

docstring

added ref

todo

changed

wallclock

commented

removed funcy

Deleted funcy,

updated configar paser that has some fixes in changed files

Improved the run/monitor speed.

Fixed some default stuff

fix stats

Some memory changes introduced

added more cases

reformat

Added test_dependencies

changed the location

re-added marked_status

File parameter

reviewing

changed results

removed root = None

update_genealogy clean unused code

update_genealogy clean unused code

reviewing comments

reviewing comments

reviewing comments

tests

tes

fix pipeline

test fix

test fix

added funcy to setup.py

updated test

changed configparserversion

better detection if data is changed

working, added the real configuration to the docs

changed configparserversion

working?

changed test

working?

issue_with_none

Added -f flag to force the recreation from 0 ... (useful mainly for test )

maybe almost working

fixed bug with chunk

wrapper fix

comments

comments

comments

comments

comments

comments

doble #

job_section

comments

docstring

added ref

todo

changed

wallclock

commented

removed funcy

Deleted funcy,

updated configar paser that has some fixes in changed files

Improved the run/monitor speed.

Fixed some default stuff

fix stats

Some memory changes introduced

reviewing changes (comments)

reviewing changes (comments)

reviewing changes (comments)

reviewing changes (graph enumerate)

reviewing changes ( delete commentS)

reviewing changes ( delete valid parents)

reviewing changes

reviewing changes

reviewing changes

reviewing changes

reviewing changes

reviewing changes (numpy)

reviewing changes (numpy)

reviewing changes ( docstring)

reviewing changes ( docstring)

reviewing changes

reviewing changes

reviewing changes

reviewing changes

added more cases

reformat

Added test_dependencies

changed the location

re-added marked_status

File parameter

reviewing

changed results

removed root = None

update_genealogy clean unused code

update_genealogy clean unused code

reviewing comments

reviewing comments

reviewing comments

tests

tes

fix pipeline

test fix

test fix

added funcy to setup.py

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

fixing Bruno review comments

Merge lastest changes
Fixed ext header to work under this version
Fixed default type

[rocrate] Add RO-Crate support to Autosubmit.

This commit includes work from several other commits, squashed.
It started around February 2023, and by July 2023 it was validated
by the RO-Crate community, thanks especially to Simone Leo.

Unit tests and documentation were added as well. It add support
to the following three RO-Crate profiles in Autosubmit:

- Process Run Crate
- Workflow Run Crate
- Workflow RO-Crate profile 1.0

This is available through the Autosubmit commands archive
and unarchive.

revise the changes

update version

bug fix an issue with additional_files and \\ variables

added retrial key

Move temp folder to the outside of for loops to reduce file creation. Rewrite the assertion part

Add dani's check so that it doesnt complain with file not found when proj type is none

add extended header and tailer documentation

test if the file does not exist, it throws an exception

test all the routes from extended tailer and header except fetching the file

change the check of hashbang to the first two characters

Handle if user sets value with empty key

Add R, Bash, and python extended scripts

Fix an issue with retrials ( present in 4.0) found while testing a full run with templates and wrapper

Added platform_name to the variables to load before the rest, ( mainly when building the dict )

Fixed -cw in create, like in inspect

Re-adapted some test-cases to match new code

workflows fixed

fixing all workflows

fixing all workflows

fixing all workflows

                    # If parent and childs has the same amount of splits \\ doesn't make sense so it is disabled

Remove cycles ( job depends on itself)

detail is now a function

Added a local test to compare workflows from 4.0 to 4.1 using -d option

fix default values

fix split

fix split

fixed parent.split == child.split when 1//2

improved test

added get_jobs_filtered test

Improved job_list test

Improved job_list test

pipeline not working

pipeline not working

removed __eq__ due being incompatible with grand part of the code, changed the test instead

added job_list generate tests
Added __eq__
fixed an issue with dependencies None

Changed DB for PKL in tests

Added more tests

Added more tests

fix wrapper dic

added run_member test

added test_build_job_with_existent_job_list_status  test

added compare_section  test

added update_parameters  test

added update_parameters  test

added update_parameters  test

added add_child test

added _repr test

Old tests working

Only 19 remains, have to doble check grouping

fix job_list

half fix job_list

half fix job_list

fix test_job.py

fix checkpoint and doc tests

Fix member_from

more changes

numpy deleted from environment.yml

pep warning fix

added test

fix doc

docs for the new autosubmit_rc env variable

docs for the new autosubmit_rc env variable

fix doc

added another suppress

added comment

changed try: except for suppress

 -

commented the debug line

Changed version

Changes to th efunction, fix a bug with the connection, added a close for ._transport of ssh

more fixes

added a debugfunction

Added a notify for push

force portalocker to <= 2.7

removed inputtimeout from requeriments

requeriments

2fa notification change

Fix applied to 2fa, local platform may were asking for a password

Fix applied to 2fa

indent in docs

dependencies

docs

docs

added method parameter

2fa: instead of 2fa

rollback few things

2fa threads

timeout

timeout

test 2fa

added docs

CHANGED input for getpass to hide typing ( it may not work)

2fa

2fa

fix additional files for ecmwf

Fixed more issues, now edgeless nodes are correctly deleted and dependencies parameter is correctly set , fixed other issues when loading previous job_list and when the node doesnt have the job

fixed few workflow inconsistencies

fixed dependency

fixed ready jobs

more fix

Working but have an issue with the initial status

added apply_filter_1_to_1

more test

test

more fixes

bsic monitor working

working on fixing merges

working on fixing merges

Pickle working,
Futher performance improves in the manage_dependencies part

working with pickle up to 1000000, afterwards it give segfualt in saving.. looking for alternatives

MUCH faster, is probabily bugged for some cases (wip)

version update

Added a delete function for nodes that are no longer part of the workflow ( with a xor)

TODO:

Delete old nodes

Reloading only the neccesary, added two methods for asconfparser

Fix reload in create

pkl changes

working faster, no memory issues but thinking more solutions

corrected prents

testing

fast test

Fixed some bugs with refactor

More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method,  code for generate jobs run very fast, inspect working has to check other commands

Reduced uneccesary operations,
Reduced memory usage

Using igraph for perform the transitive reduction

added split filter

added split filter

setstatus refactoring

rebased

pkl changes

working faster, no memory issues but thinking more solutions

corrected prents

testing

fast test

Fixed some bugs with refactor

More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method,  code for generate jobs run very fast, inspect working has to check other commands

Reduced uneccesary operations,
Reduced memory usage

Using igraph for perform the transitive reduction

added split filter

added split filter

setstatus refactoring