# Profiling Using HDF5 User Guide

## Contents

Overview
Administration
Profiling Jobs
HDF5
Data Series

## Overview

The AcctGatherProfileType/hdf5 plugin allows SLURM to coordinate collecting data on jobs it runs on a cluster that is more detailed than is practical to include in its database. The data comes from periodically sampling various performance data either collected by SLURM, the operating system, or component software. The plugin will record the data from each source as a Time Series and also accumulate totals for each statistic for the job.

Time Series are energy data collected by an AcctGatherEnergy plugin, I/O data from a network interface collected by an AcctGatherInfiniband plugin, I/O data from parallel file systems such as Lustre, and task performance data such as local disk I/O, cpu consumption, and memory us, as well as potential data from other sources.

The data is collected into a file on a shared file system for each step on each allocated node of a job and then merged into a HDF5 file. Individual files on a shared file system was chosen because it is possible that the data is voluminous so solutions that pass data to the SLURM control daemon via RPC may not scale to very large clusters or jobs with many allocated nodes.

A seperate SLURM Profile Accounting Plugin API (AcctGatherProfileType) documents how write other Profile Accounting plugins.

## Administration

### Shared File System

The HDF5 Profile Plugin requires a common shared file system on all the compute nodes. While a job is running, the plugin writes a file into this file system for each step of the job on each node. When the job ends, the merge process is launched and the node-step files are combined into one hdf5 file for the job.

The root of the directory structure is declared in the ProfileHDF5Dir option in the acct_gather.conf file. The directory will be created by slurm if it doesn't exist.

Each user that creates a profile will have a subdirector to the profile directory that has read/write permission only for the user.

### Configuration parameters

The profile plugin is enabled in the slurm.conf file, but is internally configured in the acct_gather.conf file.

#### slurm.conf parameters

This line the slum.conf enables the HDF5 Profile Plugin.
AcctGatherProfileType=acct_gather_profile/hdf5

#### act_gather.conf parameters

There are parameters directly used by the HDF5 Profile Plugin.

ProfileHDF5Dir=<path>
  This parameter is the path to the shared folder into which the acct_gather_profile plugin will write detailed data as an HDF5 file. The directory is assumed to be on a file system shared by the controller and all compute nodes. This is a required parameter.
ProfileHDF5CollectDefault=opt{,opt{,opt}}
  Default --Profile for data types collected for each job submission. It ia a comma separated list of data streams. Use this option with caution. A node-step file will be created for on every node of every step for every job. They will not automatically be merged into job files. (Even job files for small jobs would fill the fill the file system.) This option is intended for test environments where you might want to profile a series of jobs but do not want to have to add the --profile option to the launch scripts.

#### Time Series Control Paramters

Other plugins add time series data to the HDF5 collection. They typically have a polling frequency specified in one of the above configuration files.

The following table summarized parameters that control sample frequency.

Conf file | Parameter | Time Series
--- | --- | ---
slurm.conf | JobAcctGatherFrequency | Task, Lustre
acct_gather.conf | EnergyIPMIFrequency | Energy
acct_gather.conf | InfinibandOFEDFrequency | Network

## Profiling Jobs

### Data Collection

The --profile option on salloc|sbatch|srun controls whether if data is collected and what type of data is collected. If --profile is not specified the default is no data collected (unless the ProfileHDF5CollectDefault option is used in acct_gather.conf. --profile on the command line overrides any value specified in the configuration file.)

--profile=<all|none|[energy[,|task[,|lustre[,|network]]]]>
  enables detailed data collection by the acct_gather_profile plugin. Detailed data are typically time-series that are stored in an HDF5 file for the job.

All
  All data types are collected. (Cannot be combined with other values.)

None
  No data types are collected. This is the default.
  (Cannot be combined with other values.)
Energy
  Energy data is collected.
Task
  Task (I/O, Memory, ...) data is collected.
Lustre
  Lustre data is collected.
Network
  Network (InfiniBand) data is collected.

### Data Consolidation

The node-step files are merged into one HDF5 file for the job using the sh5util.

The command line may added to the normal launch script, if the job is started with sbatch. For example;

    sbatch -n1 -d$last_job_id --wrap="sh5util --profile=none -j $last_job_id"

Note that --profile=none is required if the enclosing sbatch command included a --profile parameter.

### Data Extraction

The sh5util program can also be used to extract specific data from the hdf5 file an write it in comma separated value for importation into other analysis tools such as spreadsheets.

## HDF5

HDF5 is a well known structured data set that allows heterogeneous data but related data to be stored in one file. (.i.e. sections for energy statistics, sections for network I/O, sections for Task data, …) Its internal structure resembles a file system with groups being similar to directories and data sets being similar to files. It also allows attributes to be attached to groups to store application defined properties.

There are commodity programs, notably HDFView for viewing and manipulating these files.

Below is a screen shot from HDFView expanding the job tree and showing the attributes for a specific task.

## Data Structure

In the job file, there will be a group for each step of the job. Within each step, there will be a group for nodes, and a group for tasks.

- The nodes group will have a group for each node in the step allocation. For each node group, there is a sub-group for Time Series and another for Totals.
  - The Time Series group contains a group/dataset containing the time series for each collector.
  - The Totals group contains a corresponding group/dataset that has the Minimum, Average, Maximum, and Sum Total for each item in the time series.
- The Tasks group will only contain a subgroup for each task. It primarily contains an attribute stating the node on which the task was executed. This set of groups is essentially a cross reference table.

### Energy Data

AcctGatherEnergyType=acct_gather_energy/ipmi is required in slurm.conf to collect energy data. Also appropriately set EnergyIPMIFrequency in acct_gather.conv

Date Time
  Time of day at which the data sample was taken. This can be used to correlate activity with other sources such as logs.
Time
  Elapsed time since the begining of the step.
Power
  Power consumption during the interval.
CPU Frequency
  CPU Frequency at time of sample in kilohertz.

### Infiniband Data

JobAcctInfinibandType=acct_gather_infiniband/ofed is required in slurm.conf to collect task data. Also appropriately set InfinibandOFEDFrequency in acct_gather.conf Each data sample in the Lustre Time Series contains the following data items.

Date Time
  Time of day at which the data sample was taken. This can be used to correlate activity with other sources such as logs.
Time
  Elapsed time since the begining of the step.
Packets In
  Number of packets coming in.
Megabytes Read
  Number of megabytes coming in through the interface.
Packets Out
  Number of packets going out.
Megabytes Write
  Number of megabytes going out through the interface.

### Luster Data

JobAcctGatherType=jobacct_gather/linux is required in slurm.conf to collect task data. Also appropriately set JobAcctGatherFrequency in slurm.conf

Each data sample in the Lustre Time Series contains the following data items.

Date Time
  Time of day at which the data sample was taken. This can be used to correlate activity with other sources such as logs.
Time
  Elapsed time since the begining of the step.
Reads
  Number of read operations.
Megabytes Read
  Number of megabytes read.
Writes
  Number of write operations.
Megabytes Write
  Number of megabytes written.

### Task Data

JobAcctGatherType=jobacct_gather/linux is required in slurm.conf to collect task data. Also appropriately set JobAcctGatherFrequency in slurm.conf

Date Time
  Time of day at which the data sample was taken. This can be used to correlate activity with other sources such as logs.
Time
  Elapsed time since the begining of the step.
CPU Frequency
  CPU Frequency at time of sample.
CPU Time
  Seconds of CPU time used during the sample.
CPU Utilization
  CPU Utilization during the interval.
RSS
  Value of RSS at time of sample.
VM Size
  Value of VM Size at time of sample.
Pages
  Pages used in sample.
Read Megabytes
  Number of megabytes read from local disk.
Write Megabytes
  Number of megabytes written to local disk.

Last modified 17 May 2013