|
|
|
## Setting up Operational Workflows on Jenkins (BSC-ES)
|
|
|
|
|
|
|
|
To set up an operational SUNSET workflow, contact @vagudets. This wiki documents the steps of the process:
|
|
|
|
|
|
|
|
### Step 1. Create a script and a recipe template for the operational workflow [For users]
|
|
|
|
|
|
|
|
1. Create the recipe you will use for the operational (sdate field can be left empty), and the script to be run.
|
|
|
|
2. Create a clone of the SUNSET repository in /esarchive, pointing to the commit/branch that your workflow should run from.
|
|
|
|
3. Create an autosubmit experiment to assign to this operational. See: [How to create an Autosubmit Experiment](https://autosubmit.readthedocs.io/en/master/userguide/create/index.html#create-new-experiment)
|
|
|
|
|
|
|
|
### Step 2. Create and configure a new project [For maintainers]
|
|
|
|
|
|
|
|
#### Project creation:
|
|
|
|
|
|
|
|
On the Dashboard, select "New Item", and then "Freestyle project". Give the project a name and a description.
|
|
|
|
|
|
|
|
#### Project configuration:
|
|
|
|
|
|
|
|
1. **Description**: Add the following information in the Description box:
|
|
|
|
> Operational seasonal forecast for \<Project Name>
|
|
|
|
Main researcher in charge: Name (@\<username>) # This is the person responsible for the workflow in the project
|
|
|
|
Main technician in charge: Name (@\<username>) # This is the person from CES responsible for the Jenkins integration of the operational
|
|
|
|
[ Any other relevant details ]
|
|
|
|
|
|
|
|
2. **Enable project-based security**: The person in charge and any other people who may need to run the workflow should be given permissions to run the project here.
|
|
|
|
3. **Discard old builds?**: This option enables automatic log deletion. Logs can be deleted after X number of days, or after X number of builds (for example: keep only logs for the past 50 builds). It is recommended to activate this option in order to save disk space, keeping in mind that the job logs will already be stored in the output directory set in the SUNSET recipe.
|
|
|
|
|
|
|
|
### Step 3. Create all necessary credentials on Jenkins [For users]
|
|
|
|
|
|
|
|
This step only needs to be done once for each user. Credentials may need to be updated if there is a password change.
|
|
|
|
|
|
|
|
1. **Add credentials for user here:** https://earth.bsc.es/jenkins/credentials/store/system/domain/_/. Your general BSC username and password (the ones you use to log into the Hub, calendar, wiki...) are needed. The HPC credentials are not needed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. **Register connection with user + machine (e.g. bsceshub0x) here:** https://earth.bsc.es/jenkins/configure. At the end of the "SSH remote hosts" section, click on the Add button and fill in the Hostname for the and select your credentials. You can click on "Check connection" to make sure it works correctly. Once the connection has been registered, remember to save!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Step 4. Modify the project configuration
|
|
|
|
|
|
|
|
Once the credentials and access to the machines is set up, it is time to configure when and how the project will run.
|
|
|
|
|
|
|
|
1. **Build Triggers -> Build Periodically -> Schedule**. With this option, we can schedule the workflow to run periodically at a certain time. We can choose the minute, hour, and day of the month or week. Clicking on the "?" next to the "Schedule" box will show a detailed explanation of the scheduling syntax and the different possibilities.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. **Build -> Add build step -> Execute shell script on remote host using ssh**. Select the previously created ssh connection for "SSH Site" and add the call to SUNSET-oper.sh in "Command". The command should have the following structure:
|
|
|
|
|
|
|
|
```shell
|
|
|
|
bash <path_to_code_directory>/SUNSET-oper.sh --expid=<expid> --recipe=<path_to_recipe> --horizon=<subseasonal/seasonal> --code_dir=<path_to_code_directory> &>> <log_file>
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3. **Post-build Actions -> Add post-build action -> E-mail notification**. Add the person in charge of the operational and any relevant people to the list of recipients and activate the "Send e-mail for every unstable build" option.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
 |
|
|
\ No newline at end of file |
