Simple Linux Utility for Resource Management

Home

About
Overview
What's New
Publications
SLURM Team

Using
Documentation
FAQ
Getting Help
Mailing Lists

Installing
Platforms
Download
Guide

SLURM Node Selection Plugin API

Overview

This document describes SLURM node selection plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own SLURM node selection plugins. This is version 0 of the API.

SLURM node selection plugins are SLURM plugins that implement the SLURM node selection API described herein. They are intended to provide a mechanism for both selecting nodes for pending jobs and performing any system-specific tasks for job launch or termination. The plugins must conform to the SLURM Plugin API with the following specifications:

const char plugin_type[]
The major type must be "select." The minor type can be any recognizable abbreviation for the type of node selection algorithm. We recommend, for example:

  • linear—A plugin that selects nodes assuming a one-dimensional array of nodes. The nodes are selected so as to minimize the number of consecutive sets of nodes utilizing a best-fit algorithm. While supporting shared nodes, this plugin does not allocate individual processors, memory, etc. This plugin is recommended for systems without shared nodes.
  • cons_res—A plugin that can allocate individual processors, memory, etc. within nodes. This plugin is recommended for systems with many non-parallel programs sharing nodes. For more information see Consumable Resources in SLURM.
  • bluegeneIBM Blue Gene node selector. Note that this plugin not only selects the nodes for a job, but performs some initialization and termination functions for the job.

The plugin_name and plugin_version symbols required by the SLURM Plugin API require no specialization for node selection support. Note carefully, however, the versioning discussion below.

A simplified flow of logic follows:

slurmctld daemon starts
if (select_p_state_restor)() != SLURM_SUCCESS)
   abort

slurmctld reads the rest of its configuration and state information
if (select_p_node_init() != SLURM_SUCCESS)
   abort
if (select_p_part_init() != SLURM_SUCCESS)
   abort

wait for job 
if (select_p_job_test(all available nodes) != SLURM_SUCCESS) {
   if (select_p_job_test(all configured nodes) != SLURM_SUCCESS)
      reject the job and tell the user it can never run
   else
      leave the job queued for later execution
} else {
   update job's node list and node bitmap
   if (select_p_job_begin() != SLURM_SUCCESS)
      leave the job queued for later execution
   else {
      while (!select_p_job_ready())
        wait
      execute the job
      wait for job to end or be terminated
      select_p_job_fini()
    }
}

wait for slurmctld shutdown request
select_p_state_save()
Depending upon failure modes, it is possible that select_p_state_save() will not be called at slurmctld terminatation. When slurmctld is restarted, other function calls may be replayed. select_p_node_init() may be used to syncronize the plugin's state with that of slurmctld.

Data Objects

These functions are expected to read and/or modify data structures directly in the slurmctld daemon's memory. Slurmctld is a multi-threaded program with independent read and write locks on each data structure type. Thererfore the type of operations permitted on various data structures is identified for each function.

These functions make use of bitmaps corresponding to the nodes in a table. The function select_p_node_init() should be used to establish the initial mapping of bitmap entries to nodes. Functions defined in src/common/bitmap.h should be used for bitmap manipulations (these functions are directly accessible from the plugin).

API Functions

The following functions must appear. Functions which are not implemented should be stubbed.

Global Node Selection Functions

int select_p_state_save (char *dir_name);

Description: Save any global node selection state information to a file within the specified directory. The actual file name used is plugin specific. It is recommended that the global switch state contain a magic number for validation purposes. This function is called by the slurmctld deamon on shutdown.

Arguments: dir_name    (input) fully-qualified pathname of a directory into which user SlurmUser (as defined in slurm.conf) can create a file and write state information into that file. Cannot be NULL.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

int select_p_state_restore (char *dir_name);

Description: Restore any global node selection state information from a file within the specified directory. The actual file name used is plugin specific. It is recommended that any magic number associated with the global switch state be verified. This function is called by the slurmctld deamon on startup.

Arguments: dir_name    (input) fully-qualified pathname of a directory containing a state information file from which user SlurmUser (as defined in slurm.conf) can read. Cannot be NULL.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR, causing slurmctld to exit.

int select_p_node_init (struct node_record *node_ptr, int node_cnt);

Description: Note the initialization of the node record data structure. This function is called when the node records are initially established and again when any nodes are added to or removed from the data structure.

Arguments:
node_ptr   (input) pointer to the node data records. Data in these records can read. Nodes deleted after initiialization may have their the name field in the record cleared (zero length) rather than rebuilding the node records and bitmaps.
node_cnt    (input) number of node data records.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR, causing slurmctld to exit.

int select_p_part_init (List part_list);

Description: Note the initialization of the partition record data structure. This function is called when the partition records are initially established and again when any partition configurations change.

Arguments: part_list   (input) list of partition record entries. Note that some of these partitions may have no associated nodes. Also consider that nodes can be removed from one partition and added to a different partition.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR, causing slurmctld to exit.

int select_p_pack_node_info (time_t last_query_time, Buf *buffer_ptr);

Description: pack node specific information into a buffer.

Arguments: last_query_time   (input) time that the data was last saved. If it has not changed since this time, return SLURM_NO_CHANGE_IN_DATA.
buffer_ptre   (input/output) buffer into which the node data is appended.

Returns: SLURM_SUCCESS if successful, SLURM_NO_CHANGE_IN_DATA if data has not changed since last packed, otherwise SLURM_ERROR

Job-Specific Node Selection Functions

int select_p_job_test (struct job_record *job_ptr, bitstr_t *bitmap, int min_nodes, int max_nodes);

Description: Given a job's scheduling requirement specification and a set of nodes which might be used to satisfy the request, identify the nodes which "best" satify the request. Note that nodes being considered for allocation to the job may include nodes already allocated to other jobs, even if node sharing is not permitted. This is done to ascertain whether or not job may be allocated resources at some later time (when the other jobs complete). This permits SLURM to reject non-runnable jobs at submit time rather than after they have spent hours queued. Informing users of problems at job submission time permits them to quickly resubmit the job with appropriate constraints.

Arguments:
job_ptr    (input) pointer to the job being considered for scheduling. Data in this job record may safely be read. Data of particular interst include details->contiguous (set if allocated nodes should be contiguous), num_procs (minimum processors in allocation) and details->req_node_bitmap (specific required nodes).
bitmap    (input/output) bits representing nodes which might be allocated to the job are set on input. This function should clear the bits representing nodes not required to satisfy job's scheduling request. Bits left set will represent nodes to be used for this job. Note that the job's required nodes (details->req_node_bitmap) will be a superset bitmap when the function is called.
min_nodes    (input) minimum number of nodes to allocate to this job. Note this reflects both job and partition specifications.
max_nodes    (input) maximum number of nodes to allocate to this job. Note this reflects both job and partition specifications.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR and future attempts may be made to schedule the job.

int select_p_job_begin (struct job_record *job_ptr);

Description: Note the initiation of the specified job is about to begin. This function is called immediately after select_p_job_test() sucessfully completes for this job.

Arguments: job_ptr    (input) pointer to the job being initialized. Data in this job record may safely be read or written. The nodes and node_bitmap fields of this job record identify the nodes which have already been selected for this job to use. For an example of a job record field that the plugin may write into, see select_id.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR, which causes the job to be requeued for later execution.

int select_p_job_ready (struct job_record *job_ptr);

Description: Test if resources are configured and ready for job execution. This function is only used in the job prolog for BlueGene systems to determine if the bglblock has been booted and is ready for use.

Arguments: job_ptr    (input) pointer to the job being initialized. Data in this job record may safely be read. The nodes and node_bitmap fields of this job record identify the nodes which have already been selected for this job to use.

Returns: 1 if the job may begin execution, 0 otherwise.

int select_p_job_fini (struct job_record *job_ptr);

Description: Note the termination of the specified job. This function is called as the termination process for the job begins (prior to killing the tasks).

Arguments: job_ptr    (input) pointer to the job being terminated. Data in this job record may safely be read or written. The nodes and/or node_bitmap fields of this job record identify the nodes which were selected for this job to use.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

Get/set plugin information

int select_p_get_extra_jobinfo(struct node_record *node_ptr, struct job_record *job_ptr, enum select_data_info info, void *data);

Description: Get plugin-specific information related to the specified job and/or node.

Arguments:
node_ptr    (input) pointer to the node for which information is requested.
job_ptr    (input) pointer to the job for which information is requested.
info    (input) identifies the type of data requested.
data    (output) the requested data.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

int select_p_get_select_nodeinfo(struct node_record *node_ptr, enum select_data_info info, void *data);

Description: Get plugin-specific information related to the specified node.

Arguments:
node_ptr    (input) pointer to the node for which information is requested.
info    (input) identifies the type of data requested.
data    (output) the requested data.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

int select_p_update_nodeinfo(struct node_record *node_ptr, enum select_data_info info);

Description: Update plugin-specific information related to the specified node.

Arguments:
node_ptr    (input) pointer to the node for which information is requested.
info    (input) identifies the type of data requested.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

int select_p_get_info_from_plugin(enum select_data_info info, void *data);

Description: Get plugin-specific information.

Arguments:
info    (input) identifies the type of data to be updated.
data    (output) the requested data.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

int select_p_job_init(List job_list);

Description: Used at slurm startup to syncrhonize plugin (and node) state with that of currectly active jobs.

Arguments: job_list    (input) list of slurm jobs from slurmctld job records.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

Versioning

This document describes version 0 of the SLURM node selection API. Future releases of SLURM may revise this API. A node selection plugin conveys its ability to implement a particular API version using the mechanism outlined for SLURM plugins. In addition, the credential is transmitted along with the version number of the plugin that transmitted it. It is at the discretion of the plugin author whether to maintain data format compatibility across different versions of the plugin.


For information about this page, contact slurm-dev@lists.llnl.gov.