# Performance options

([Main README](README.md))

There are many compiler flags and environment variables that can be useful for tweaking the performance of GPU applications. You will find a couple down below.

## Flags

**-O3**  
Default of -O2 is most tested and around as good as -O3 or -fast for other compilers.

- O3 only gives slightly more than default -O2

**-hfp3**  
Gives a lot more floating point optimization.

- Default is -hfp2

**-hflex_mp=\<option\>**  
Control the aggressiveness of optimizations (e.g. FP optimizations), harder option is less reproducible!

- Value intolerant: highest reproducibility, highest performance penalty
- Value strict: use some safe optimizations, high reproducibility chance
- Value conservative: more aggressive optimization, higher performance but less suitable for repeatable applications
- Value default: more aggressive optimization, not sufficiently repeatable
- Value tolerant: most aggressive optimization, highest performance but not sufficiently repeatable

**-h restrict=[a|f]**  
Tells compiler to treat certain classes of pointers as restricted pointers.

- Enhances optimizations, like vectorization

**-h cacheN**  
Specifies level of automatic cache management to perform (N between 0 and 3). Default for N is 2.

- Value 0: cache blockign turned off
- Value 3: aggressive automatic cache management.

**-h Pic**  
Generate position independent code (PIC)

**-h[system|default]_alloc**  
The -hsystem_alloc causes the compiler to use native malloc implementation of OS. By default, compiler uses modified malloc implementation which offers better support for Cray memory needs.

## Environment variables

**MPICH_GPU_SUPPORT_ENABLED=1**  
Allow GPU resident buffers to be used in MPI calls

**MPICH_GPU_MANAGED_MEMORY_SUPPORT_ENABLED=1**  
Allows MPI to use memory addresses that can be migrated (i.e. cuda/hipMallocManaged memory).

**MPICH_OFI_NIC_POLICY=GPU**  
Selects the NIC that is closest to the GPU (GCD) to be used.

- Use MPICH_OFI_NIC_POLICY=NUMA for non-GPU codes (can also work for GPU codes)

**FI_MR_CACHE_MONITOR=memhooks**  
Forces a different memory registration cache monitor that tends to work better in authors experience.

- Usefull when encountering unexplained performance issues or MPI problems

**FI_CXI_RX_MATCH_MODE=software**  
Used when having flow control or queue exhaustion problems.

**FI_CXI_REQ_BUF_SIZE, FI_CXI_REQ_BUF_MIN_POSTED, FI_CXI_DEFAULT_CQ_SIZE**  
Change amount of buffer space available in software mathing mode.

- Increasing can OOM the job!

**MPICH_GPU_IPC_THRESHOLD=\<value\>**  
Performance tunable parameter for controlling when GPU DMA engines are used for intra-node GPU-GPU transfers.

- Messages smaller than threshold won't spend time setting up the DMA engine.
- If code does NOT reuse GPU-GPU buffers, might be worth increasing threshold from the 1024-byte default.
- But try reuse buffers, and change this parameter accordingly.
