PARALLEL DIRECTIVES
===================

teams:
    - Fork/join model (omp parallel)
    - No collective synchronization! (i.e. barriers)
    - Must be within "omp target"
    -> Code runs on GPU using multiple "teams"
    
    
distribute:
    - Loop partitioning (like omp do/far)
    - Iterations are partitioned across "teams"
    - NO implied barrier at end of loop!
    - Best practice to combine with teams (target teams distribute)
    -> Iterations of loop are partitioned across "teams".
    

! NOTE - parallel and simd are inconsistent across implementations !

parallel for / simd:
    - Used for two-level GPU parallelism
    - "teams" maps to GPU threadblocks
    - "parallel / simd" maps to GPU threads

    - CCE-Classic maps "simd" to GPU threads and skips "parallel for"
    - Clang maps "parallel for" to GPU threads and skips "simd"


do/for


barrier


MEMORY DIRECTIVES
=================
data map(<var>):
    - Maps a variable to and from host.
data map(from:<var>):
    - Maps a variable from host.
data map(to:<var>):
    - Maps a variable to host.
data map(alloc:<var>):
    - Allocate space for the variable on device.


   
STEP-BY-STEP GUIDE
==================
1) Offload code to GPU with "omp target"
 - Does not express any parallelism
2) Setup fork/join parallelism with "omp parallel"
3) Partition loops amoung workgroups with "omp distribute"
4) Partition a portion of a loop amoung threads with "parallel/simd"
5) Add data regions with "data map"
6) Optimize results with clauses (e.g. )



DEBUG OPTIONS
=============
-hmsgs -hlist=m flags
    - Indicate what regions are offloaded.

CRAY_ACC_DEBUG envvar
    - Emits messges for all offloading operations
    - Value 1: Concise info on offloading regions
    - Value 2: More in-depth runtime info, but user friendly.
    - Value 3: Very verbose (i.e. memory addresses), not designed for everyday user.
    
LIBOMPTARGET_INFO envvar
    - Controls what's printed, but not how much
    - Value is a bitfield: https://openmp.llvm.org/design/Runtimes.html#libomptarget-info
    - Set to -1 to get it all.
    
LIBOMPTARGET_DEBUG envvar
    - Like the one above, but much more info (probably too much)
    
AMD_SERIALIZE_KERNEL envvar
    - Good for checking kernel race conditions.
    - Value 1: Synchronize before launces (i.e. make sure everything is done on GPU)
    - Value 2: Synchronize after launces (i.e. wait for kernel to finish before moving one)
    - Value 3: Do both 1 and 2

AMD_SERIALIZE_COPY envvar
    - Good for checking kernel race conditions.
    - Value 1: Synchronize before copies (i.e. make sure everything is done on GPU)
    - Value 2: Synchronize after copies (i.e. wait for kernel to finish before moving one)
    - Value 3: Do both 1 and 2
    
MPICH_ENV_DISPLAY=1
    - Logs exactly how MPI was configured
    
MPICH_VERSION_DISPLAY=1
    - Helps to track how issues change with different MPI versions
    
    
! NOTE: DO NOT USE THE ONES BELOW IN PRODUCTION !
    
MPICH_SINGLE_HOST_ENABLED=0
    - Force traffic through the NIC when you only have one node
    
MPICH_SMP_SINGLE_COPY_MODE=NONE
    - Disables all intranode optimizations and forces all in-node communication through slow interprocess communicator paths on CPU memory.
    
MPICH_GPU_IPC_ENABLED=0
    - Disabling SMP operations, also disables IPC.



PERFORMANCE OPTIONS
===================
MPICH_GPU_SUPPORT_ENABLED=1
    - Allow GPU resident buffers to be used in MPI calls
    
MPICH_GPU_MANAGED_MEMORY_SUPPORT_ENABLED=1
    - Allows MPI to use memory addresses that can be migrated (i.e. cuda/hipMallocManaged memory)

MPICH_OFI_NIC_POLICY=GPU
    - Selects the NIC that is closest to the GPU (GCD) bein used
    -> Use MPICH_OFI_NIC_POLICY=NUMA for non-GPU codes (can also work for GPU codes)
    
FI_MR_CACHE_MONITOR=memhooks
    - Forces a different memory registration cache monitor that tends to work better in authors experience
    - Usefull when encountering unexplained performance issues or MPI problems

FI_CXI_RX_MATCH_MODE=software
    - Used when having flow control or queue exhaustion problems.
    
FI_CXI_REQ_BUF_SIZE, 
FI_CXI_REQ_BUF_MIN_POSTED,
FI_CXI_DEFAULT_CQ_SIZE,
    - Change amount of buffer space available in software mathing mode
    - Increasing can OOM the job!
    
MPICH_GPU_IPC_THRESHOLD=<value>
    - Performance tunable parameter for controlling when GPU DMA engines are used for intra-node GPU-GPU transfers
    - Messages smaller than threshold won't spend time setting up the DMA engine.
    - If code does NOT reuse GPU-GPU buffers, might be worth increasing threshold from the 1024-byte default.
    - But try reuse buffers, and change this parameter accordingly.
