# Setup

The supercomputers require different setups for running the offloaded code. This section will cover the used supercomputers, how to setup your environment correctly, and how to compile code correctly. Refer to the [coding](coding) folder for info on the programming aspect, and to the [profiling](profiling) folder for info on profiling.

## Lumi

### Reserving a compute node

In order to allocate a GPU compute node, it is required to specify the `standard-g` partition. It is not possible to ssh into compute nodes, thus one has to allocate a node and then run bash onto that node ***within the same terminal session***.

For allocating an interactive node, use:

```bash
salloc \
    --gpus-per-node=8 \
    --exclusive \
    --nodes=1 \
    --ntasks=8 \
    --partition=standard-g \
    --account=project_465000454 \
    --time=7:40:00 \
    --mem=0 \
    --job-name=interactive_node
```

Then Bash onto the node with:

```bash
srun --cpu_bind=none --nodes=1 --pty bash -i
```

It might be handy to create aliases for these commands in the `~/.bashrc`.

### Loading modules

In order to perform computations on the node, you need to load the folowing modules:

```bash
module load LUMI/23.03 partition/G PrgEnv-cray cpe/23.03 craype-x86-trento craype-accel-amd-gfx90a rocm cray-fftw/3.3.10.1 cray-hdf5/1.12.1.5 cray-netcdf/4.8.1.5
```

### Compiler

For compiling code with AMD GPU offloaded code, we use the `ftn` compiler. Commonly used flags are: `-O3 -homp -hnoacc -Ofast -fopenmp -lm`

## MeluXina

### Reserving a compute node

In order to allocate a GPU compute node, it is required to specify the `gpudev` reservation. For allocating an interactive node, use:

```bash
salloc --qos=dev --reservation=gpudev -N1 -A p200177 -t360
```

After allocating a node, the system will automatically SSH you onto the compute node. You can exit with `exit` and ssh back into the node afterwards.

It might be handy to create an alias for this command in the `~/.bashrc`.

### Loading modules

In order to perform computations on the node, you need to load the folowing modules:

```bash
module load env/staging/2022.1 NVHPC/22.7
```

### Compiler

For compiling code with AMD GPU offloaded code, we use the `nvfortran` compiler. Commonly used flags are: `-O3 -fopenmp -fast -mp=gpu -lm`. Other commonly used flags can be found [here](https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#freq-used-options).

## MareNostrum 4

TODO
