- 20 Mar, 2024 27 commits
-
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
Fix trace when exception as e is not an AutosubmitCritical or AutosubmitError update version
-
dbeltran authored
Fix trace when exception as e is not an AutosubmitCritical or AutosubmitError update version
-
dbeltran authored
Fix trace when exception as e is not an AutosubmitCritical or AutosubmitError
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
Added retrials for recovery
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
-
dbeltran authored
Fixed issues related to clena_runtime_parameters # deleted
-
dbeltran authored
-
dbeltran authored
-
- 05 Mar, 2024 1 commit
-
-
dbeltran authored
added autokey word # pending some values testproof added autokey word # pending some values testproof work on going calendar for splits calendar calendar calendar tentative split Changed blank message to warning standarized on_submission debug line configparser version outdated Fix issues with totaljobs, check_script Clean outdate code added a missing counter for totaljobs added a missing counter for totaljobs import change location simplified send_command simplified if remove unused array Fix monitor not showing the correct status of the workflow when -cw is prompt clean code keys_to_erase regex changed configparser version Fixed the "%CURRENT_*% issue Fixed the "%CURRENT_*% issue added job.packed = False once the job is ready or completed/failed (affects reruns) fixed queue and partition not being updated properly delete_edgeless_jobs changed changed == for is to evaluate the type removed get_all_filter_jobs changed _create_jobs_splits Now, if -cw is used in monitor or inspect. It won't check all the templates. Update configparser version to fix load/save Fixed a perfomance issue Fixed an issue with running: once adding more edges fix run Clean the code a bit working now needs some cleaning test fix bug fix bugs, reworked a bit fix bug updated test updated test fixed some bugs, added some docs more fixes test fix pipeline fix pipeline math readded datetime readded fix issue with delay retrial fix issue with -1 Fixes !#1209 fix grouping test test fix test fix some changes for Bruno comments moved if inside gen Workflow optimizations added ( mega squashed commit ) Fixes #1158 added zipp dependency ( rocrate in bscearth000) re-added additional files Database is locked error in historic db ( I think it is an issue in my computer as happened in master aswell) QOL when splits is introduced with "" ( testing francesc experiment ) Ran regression test, noticed issue with experiment a005 and fixed converse job to list changed == for in to dont care about spaces Fix splits when * and not * is in same line added if not monitor Fix changes Fix delay Fixed edge_info Differences fixed Differences fixed comments fixed comments added comments added N-1 deleted test of deleted function deleted old code fixed pipeline Fixed save Added version and hpcarch as requisites to change Improved split_to Improved split_to (wip) Added "previous" filter (wip) Added "previous" filter fixed status .lower() added Add filter previous docs python3 or pytho2 ( fixed) type python updated test changed configparserversion better detection if data is changed working, added the real configuration to the docs changed configparserversion working? changed test working? issue_with_none Added -f flag to force the recreation from 0 ... (useful mainly for test ) maybe almost working fixed bug with chunk wrapper fix comments comments comments comments comments comments doble # job_section comments docstring added ref todo changed wallclock commented removed funcy Deleted funcy, updated configar paser that has some fixes in changed files Improved the run/monitor speed. Fixed some default stuff fix stats Some memory changes introduced added more cases reformat Added test_dependencies changed the location re-added marked_status File parameter reviewing changed results removed root = None update_genealogy clean unused code update_genealogy clean unused code reviewing comments reviewing comments reviewing comments tests tes fix pipeline test fix test fix added funcy to setup.py updated test changed configparserversion better detection if data is changed working, added the real configuration to the docs changed configparserversion working? changed test working? issue_with_none Added -f flag to force the recreation from 0 ... (useful mainly for test ) maybe almost working fixed bug with chunk wrapper fix comments comments comments comments comments comments doble # job_section comments docstring added ref todo changed wallclock commented removed funcy Deleted funcy, updated configar paser that has some fixes in changed files Improved the run/monitor speed. Fixed some default stuff fix stats Some memory changes introduced reviewing changes (comments) reviewing changes (comments) reviewing changes (comments) reviewing changes (graph enumerate) reviewing changes ( delete commentS) reviewing changes ( delete valid parents) reviewing changes reviewing changes reviewing changes reviewing changes reviewing changes reviewing changes (numpy) reviewing changes (numpy) reviewing changes ( docstring) reviewing changes ( docstring) reviewing changes reviewing changes reviewing changes reviewing changes added more cases reformat Added test_dependencies changed the location re-added marked_status File parameter reviewing changed results removed root = None update_genealogy clean unused code update_genealogy clean unused code reviewing comments reviewing comments reviewing comments tests tes fix pipeline test fix test fix added funcy to setup.py fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments Merge lastest changes Fixed ext header to work under this version Fixed default type [rocrate] Add RO-Crate support to Autosubmit. This commit includes work from several other commits, squashed. It started around February 2023, and by July 2023 it was validated by the RO-Crate community, thanks especially to Simone Leo. Unit tests and documentation were added as well. It add support to the following three RO-Crate profiles in Autosubmit: - Process Run Crate - Workflow Run Crate - Workflow RO-Crate profile 1.0 This is available through the Autosubmit commands archive and unarchive. revise the changes update version bug fix an issue with additional_files and \\ variables added retrial key Move temp folder to the outside of for loops to reduce file creation. Rewrite the assertion part Add dani's check so that it doesnt complain with file not found when proj type is none add extended header and tailer documentation test if the file does not exist, it throws an exception test all the routes from extended tailer and header except fetching the file change the check of hashbang to the first two characters Handle if user sets value with empty key Add R, Bash, and python extended scripts Fix an issue with retrials ( present in 4.0) found while testing a full run with templates and wrapper Added platform_name to the variables to load before the rest, ( mainly when building the dict ) Fixed -cw in create, like in inspect Re-adapted some test-cases to match new code workflows fixed fixing all workflows fixing all workflows fixing all workflows # If parent and childs has the same amount of splits \\ doesn't make sense so it is disabled Remove cycles ( job depends on itself) detail is now a function Added a local test to compare workflows from 4.0 to 4.1 using -d option fix default values fix split fix split fixed parent.split == child.split when 1//2 improved test added get_jobs_filtered test Improved job_list test Improved job_list test pipeline not working pipeline not working removed __eq__ due being incompatible with grand part of the code, changed the test instead added job_list generate tests Added __eq__ fixed an issue with dependencies None Changed DB for PKL in tests Added more tests Added more tests fix wrapper dic added run_member test added test_build_job_with_existent_job_list_status test added compare_section test added update_parameters test added update_parameters test added update_parameters test added add_child test added _repr test Old tests working Only 19 remains, have to doble check grouping fix job_list half fix job_list half fix job_list fix test_job.py fix checkpoint and doc tests Fix member_from more changes numpy deleted from environment.yml pep warning fix added test fix doc docs for the new autosubmit_rc env variable docs for the new autosubmit_rc env variable fix doc added another suppress added comment changed try: except for suppress - commented the debug line Changed version Changes to th efunction, fix a bug with the connection, added a close for ._transport of ssh more fixes added a debugfunction Added a notify for push force portalocker to <= 2.7 removed inputtimeout from requeriments requeriments 2fa notification change Fix applied to 2fa, local platform may were asking for a password Fix applied to 2fa indent in docs dependencies docs docs added method parameter 2fa: instead of 2fa rollback few things 2fa threads timeout timeout test 2fa added docs CHANGED input for getpass to hide typing ( it may not work) 2fa 2fa fix additional files for ecmwf Fixed more issues, now edgeless nodes are correctly deleted and dependencies parameter is correctly set , fixed other issues when loading previous job_list and when the node doesnt have the job fixed few workflow inconsistencies fixed dependency fixed ready jobs more fix Working but have an issue with the initial status added apply_filter_1_to_1 more test test more fixes bsic monitor working working on fixing merges working on fixing merges Pickle working, Futher performance improves in the manage_dependencies part working with pickle up to 1000000, afterwards it give segfualt in saving.. looking for alternatives MUCH faster, is probabily bugged for some cases (wip) version update Added a delete function for nodes that are no longer part of the workflow ( with a xor) TODO: Delete old nodes Reloading only the neccesary, added two methods for asconfparser Fix reload in create pkl changes working faster, no memory issues but thinking more solutions corrected prents testing fast test Fixed some bugs with refactor More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method, code for generate jobs run very fast, inspect working has to check other commands Reduced uneccesary operations, Reduced memory usage Using igraph for perform the transitive reduction added split filter added split filter setstatus refactoring rebased pkl changes working faster, no memory issues but thinking more solutions corrected prents testing fast test Fixed some bugs with refactor More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method, code for generate jobs run very fast, inspect working has to check other commands Reduced uneccesary operations, Reduced memory usage Using igraph for perform the transitive reduction added split filter added split filter setstatus refactoring cherry picked fix ( changes in nodes declaration ) total_jobs fix suppress changed assert added unit test added unit test added unit test split the function in some smaller functions to add unit testing easier split the function in some smaller functions to add unit testing easier split the function in some smaller functions to add unit testing easier fix wrapper and totaljobs added retrial, command is failing sometimes removed submit_cmd from update_cmds added .get moved - at the end changed requeriments now all regex includes _.- update_version changes Add mailmap file (cherry picked from commit 4f76976f) added --exclusive changed the bugfix sleep rechanged fix bug Changed to pass ro_crate and be more robust to miss introductions Changed to pass ro_crate and be more robust to miss introductions Stats now counts the exact amount of processors if node is prompt Queuing times for inner_jobs fixed updated test updated test changed version .lower() added Add filter previous changed name changed name added test ( same than 4.1) version Fixes #1158 changed version Add SECURITY.MD file Add documentation about exit status of autosubmit run Return >0 if the autosubmit command fails .lower() added Add filter previous changed name changed name added test ( same than 4.1) fixed bug with chunk version wallclock Fixes #1158 Fixes #1158 added zipp dependency ( rocrate in bscearth000) changed version
-
- 04 Mar, 2024 2 commits
-
-
dbeltran authored
feedback fix added : added \n Added permissionError Traces hidden Changed pkl open Added replace instead of rename changed to not create instead of monitor Changes in save/load added --no-requeue fix set again retrials to 3 Added removal of duplicated jobs test platform job_id removal added placeholder reset fail_count on setstatus from failed to X and if graph is new reset fail_count on setstatus from failed to X and if graph is new Changed a test config , fixed issue with workflow generation Changed a test config , fixed issue with workflow generation check defaulted to False instead of True Changed blank message to warning standarized on_submission debug line configparser version outdated Fix issues with totaljobs, check_script Clean outdate code added a missing counter for totaljobs added a missing counter for totaljobs import change location simplified send_command simplified if remove unused array Fix monitor not showing the correct status of the workflow when -cw is prompt clean code keys_to_erase regex changed configparser version Fixed the "%CURRENT_*% issue Fixed the "%CURRENT_*% issue added job.packed = False once the job is ready or completed/failed (affects reruns) fixed queue and partition not being updated properly delete_edgeless_jobs changed changed == for is to evaluate the type removed get_all_filter_jobs changed _create_jobs_splits Now, if -cw is used in monitor or inspect. It won't check all the templates. Update configparser version to fix load/save Fixed a perfomance issue Fixed an issue with running: once adding more edges fix run Clean the code a bit working now needs some cleaning test fix bug fix bugs, reworked a bit fix bug updated test updated test fixed some bugs, added some docs more fixes test fix pipeline fix pipeline math readded datetime readded fix issue with delay retrial fix issue with -1 Fixes !#1209 fix grouping test test fix test fix some changes for Bruno comments moved if inside gen Workflow optimizations added ( mega squashed commit ) Fixes #1158 added zipp dependency ( rocrate in bscearth000) re-added additional files Database is locked error in historic db ( I think it is an issue in my computer as happened in master aswell) QOL when splits is introduced with "" ( testing francesc experiment ) Ran regression test, noticed issue with experiment a005 and fixed converse job to list changed == for in to dont care about spaces Fix splits when * and not * is in same line added if not monitor Fix changes Fix delay Fixed edge_info Differences fixed Differences fixed comments fixed comments added comments added N-1 deleted test of deleted function deleted old code fixed pipeline Fixed save Added version and hpcarch as requisites to change Improved split_to Improved split_to (wip) Added "previous" filter (wip) Added "previous" filter fixed status .lower() added Add filter previous docs python3 or pytho2 ( fixed) type python updated test changed configparserversion better detection if data is changed working, added the real configuration to the docs changed configparserversion working? changed test working? issue_with_none Added -f flag to force the recreation from 0 ... (useful mainly for test ) maybe almost working fixed bug with chunk wrapper fix comments comments comments comments comments comments doble # job_section comments docstring added ref todo changed wallclock commented removed funcy Deleted funcy, updated configar paser that has some fixes in changed files Improved the run/monitor speed. Fixed some default stuff fix stats Some memory changes introduced added more cases reformat Added test_dependencies changed the location re-added marked_status File parameter reviewing changed results removed root = None update_genealogy clean unused code update_genealogy clean unused code reviewing comments reviewing comments reviewing comments tests tes fix pipeline test fix test fix added funcy to setup.py updated test changed configparserversion better detection if data is changed working, added the real configuration to the docs changed configparserversion working? changed test working? issue_with_none Added -f flag to force the recreation from 0 ... (useful mainly for test ) maybe almost working fixed bug with chunk wrapper fix comments comments comments comments comments comments doble # job_section comments docstring added ref todo changed wallclock commented removed funcy Deleted funcy, updated configar paser that has some fixes in changed files Improved the run/monitor speed. Fixed some default stuff fix stats Some memory changes introduced reviewing changes (comments) reviewing changes (comments) reviewing changes (comments) reviewing changes (graph enumerate) reviewing changes ( delete commentS) reviewing changes ( delete valid parents) reviewing changes reviewing changes reviewing changes reviewing changes reviewing changes reviewing changes (numpy) reviewing changes (numpy) reviewing changes ( docstring) reviewing changes ( docstring) reviewing changes reviewing changes reviewing changes reviewing changes added more cases reformat Added test_dependencies changed the location re-added marked_status File parameter reviewing changed results removed root = None update_genealogy clean unused code update_genealogy clean unused code reviewing comments reviewing comments reviewing comments tests tes fix pipeline test fix test fix added funcy to setup.py fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments fixing Bruno review comments Merge lastest changes Fixed ext header to work under this version Fixed default type [rocrate] Add RO-Crate support to Autosubmit. This commit includes work from several other commits, squashed. It started around February 2023, and by July 2023 it was validated by the RO-Crate community, thanks especially to Simone Leo. Unit tests and documentation were added as well. It add support to the following three RO-Crate profiles in Autosubmit: - Process Run Crate - Workflow Run Crate - Workflow RO-Crate profile 1.0 This is available through the Autosubmit commands archive and unarchive. revise the changes update version bug fix an issue with additional_files and \\ variables added retrial key Move temp folder to the outside of for loops to reduce file creation. Rewrite the assertion part Add dani's check so that it doesnt complain with file not found when proj type is none add extended header and tailer documentation test if the file does not exist, it throws an exception test all the routes from extended tailer and header except fetching the file change the check of hashbang to the first two characters Handle if user sets value with empty key Add R, Bash, and python extended scripts Fix an issue with retrials ( present in 4.0) found while testing a full run with templates and wrapper Added platform_name to the variables to load before the rest, ( mainly when building the dict ) Fixed -cw in create, like in inspect Re-adapted some test-cases to match new code workflows fixed fixing all workflows fixing all workflows fixing all workflows # If parent and childs has the same amount of splits \\ doesn't make sense so it is disabled Remove cycles ( job depends on itself) detail is now a function Added a local test to compare workflows from 4.0 to 4.1 using -d option fix default values fix split fix split fixed parent.split == child.split when 1//2 improved test added get_jobs_filtered test Improved job_list test Improved job_list test pipeline not working pipeline not working removed __eq__ due being incompatible with grand part of the code, changed the test instead added job_list generate tests Added __eq__ fixed an issue with dependencies None Changed DB for PKL in tests Added more tests Added more tests fix wrapper dic added run_member test added test_build_job_with_existent_job_list_status test added compare_section test added update_parameters test added update_parameters test added update_parameters test added add_child test added _repr test Old tests working Only 19 remains, have to doble check grouping fix job_list half fix job_list half fix job_list fix test_job.py fix checkpoint and doc tests Fix member_from more changes numpy deleted from environment.yml pep warning fix added test fix doc docs for the new autosubmit_rc env variable docs for the new autosubmit_rc env variable fix doc added another suppress added comment changed try: except for suppress - commented the debug line Changed version Changes to th efunction, fix a bug with the connection, added a close for ._transport of ssh more fixes added a debugfunction Added a notify for push force portalocker to <= 2.7 removed inputtimeout from requeriments requeriments 2fa notification change Fix applied to 2fa, local platform may were asking for a password Fix applied to 2fa indent in docs dependencies docs docs added method parameter 2fa: instead of 2fa rollback few things 2fa threads timeout timeout test 2fa added docs CHANGED input for getpass to hide typing ( it may not work) 2fa 2fa fix additional files for ecmwf Fixed more issues, now edgeless nodes are correctly deleted and dependencies parameter is correctly set , fixed other issues when loading previous job_list and when the node doesnt have the job fixed few workflow inconsistencies fixed dependency fixed ready jobs more fix Working but have an issue with the initial status added apply_filter_1_to_1 more test test more fixes bsic monitor working working on fixing merges working on fixing merges Pickle working, Futher performance improves in the manage_dependencies part working with pickle up to 1000000, afterwards it give segfualt in saving.. looking for alternatives MUCH faster, is probabily bugged for some cases (wip) version update Added a delete function for nodes that are no longer part of the workflow ( with a xor) TODO: Delete old nodes Reloading only the neccesary, added two methods for asconfparser Fix reload in create pkl changes working faster, no memory issues but thinking more solutions corrected prents testing fast test Fixed some bugs with refactor More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method, code for generate jobs run very fast, inspect working has to check other commands Reduced uneccesary operations, Reduced memory usage Using igraph for perform the transitive reduction added split filter added split filter setstatus refactoring rebased pkl changes working faster, no memory issues but thinking more solutions corrected prents testing fast test Fixed some bugs with refactor More memory optimization and call optimizations, deleted uneccesary attr when generating the job becasue they will be added later with update_parameters method, code for generate jobs run very fast, inspect working has to check other commands Reduced uneccesary operations, Reduced memory usage Using igraph for perform the transitive reduction added split filter added split filter setstatus refactoring cherry picked fix ( changes in nodes declaration ) total_jobs fix suppress changed assert added unit test added unit test added unit test split the function in some smaller functions to add unit testing easier split the function in some smaller functions to add unit testing easier split the function in some smaller functions to add unit testing easier fix wrapper and totaljobs added retrial, command is failing sometimes removed submit_cmd from update_cmds added .get moved - at the end changed requeriments now all regex includes _.- update_version changes Add mailmap file (cherry picked from commit 4f76976f) added --exclusive changed the bugfix sleep rechanged fix bug Changed to pass ro_crate and be more robust to miss introductions Changed to pass ro_crate and be more robust to miss introductions Stats now counts the exact amount of processors if node is prompt Queuing times for inner_jobs fixed updated test updated test changed version .lower() added Add filter previous changed name changed name added test ( same than 4.1) version Fixes #1158 changed version Add SECURITY.MD file Add documentation about exit status of autosubmit run Return >0 if the autosubmit command fails .lower() added Add filter previous changed name changed name added test ( same than 4.1) fixed bug with chunk version wallclock Fixes #1158 Fixes #1158 added zipp dependency ( rocrate in bscearth000) changed version
- 29 Feb, 2024 2 commits
-
-
Bruno de Paula Kinoshita authored
Fix: This commit fixes an error message inconsistency See merge request !402
-
-
- 02 Jan, 2024 1 commit
-
-
Bruno de Paula Kinoshita authored
Add mailmap file See merge request !393
-
- 30 Nov, 2023 2 commits
-
-
Bruno P. Kinoshita authored
-
Bruno de Paula Kinoshita authored
Add SECURITY.md Closes #884 See merge request !363
-
- 15 Nov, 2023 1 commit
-
-
Bruno de Paula Kinoshita authored
Return >0 if the autosubmit command fails Closes #1038 See merge request !390
-
- 14 Nov, 2023 1 commit
-
-
- 12 Nov, 2023 1 commit
-
-
dbeltran authored
-
- 09 Nov, 2023 2 commits
-
-
Bruno P. Kinoshita authored
-
Bruno P. Kinoshita authored
-