SLURM message traffic summary

Below is a very terse outline of possible message traffic that may occur
across the various components of slurm. These messages will most likely
form a REQ/REPLY pair, where - following Moe's convention - the Input is
contained in the `request' message, and the Output will be found in the
`reply.'

Command(s):	Get job/accounting/node/partition/job_step/build information, 
		separate API for each data type
Client:   	squeue and scontrol commands, plus DPCS from API, any node in cluster
Server: 	slurmctld
Input:  	time-stamp, version, user id
		flags	: might be useful for filtering data sent, e.g. just this user's jobs 
Output: 	error code, version, time-stamp, record count, array of records
Notes:  	most information generally available, some might be restricted by user id


Command(s):	Get partition_key
Client:  	API call (used by DPCS)
Server: 	slurmctld
Input:  	uid (must be root)
Output: 	partition_key
Notes:   	used to control access to some partitions. for example, any user 
		can run jobs in the "batch" partition, but only when initiated by 
		a batch controller (e.g. DPCS). this prevents users from running 
		jobs outside of the queue structure


Command(s):	Allocate job
Client:   	srun or slurm api call
Server: 	slurmctld 
Input:		username/uid,nnodes,ntasks, group
		optional: partition,time_limit,constraints,features,node list, partition_key
		flags   : wait_for_resources, test only (don't allocate resources, 
		          just reply whether or not allocate would have succeeded, 
		          used by DPCS)
Output: 	job_id, return code, error code, node list, ncpus for *each* node in list, 
		job_key
Notes:  	allocate resources to a ``job''


Command(s):	Claim job allocation
Client:   	srun
Server: 	slurmctld
Input:  	uid, job_id, job_key
Output: 	error_code
Notes:  	ties allocation to a specific process_id, used to determine when a 
		job is really complete


Command(s):	Submit job
Client:   	srun or slurm api call
Server: 	slurmctld
Input:  	Allocate input + script path, environment, cwd 
                optional: partition, time_limit, constraints, features, 
		          I/O location, signal handling, partition_key
		flags:
Output: 	job_id, return code, error code
Notes:  	submit a batch job to the slurm queue


Command(s):	Register Job Step	
Client:   	srun or slurm api call
Server: 	slurmctld
Input:  	job_id,username/uid
		optional: nnodes,ntasks,cpus_per_task,distribution,time_limit,
		          constraints,features,signal handling
		flags   : wait_for_resources
Output: 	job_id, step_id, return code, error code, node list, ncpus/node
 		credential list,
Notes:  	run a set of parallel tasks under an allocated job
 		allocate resources if job_id < MIN_JOBID, otherwise assume 
		resources are already available


Command(s):	Get Job Step Information
Client:   	srun, scancel 
Server: 	slurmctld
Input:  	job_id, step_id, uid
Output: 	return code, error code, node list, ncpus/node(?)
Notes:  	Obtain the list of resources assigned to a currently executing 
                job step. Needed by at least `srun --attach`, and scancel.
		uid must match that of job_id or be root
	
	
Command(s):	Run Job Step Request
Client:   	srun or slurmctld
Server: 	slurmd
Input:  	username/uid, job_id, step_id, credential, ntasks, environment, 
		cwd, command line, stdin location, stdout/err location
Output: 	return code, error code
Notes:  	request initiation of ntasks tasks on this node.


Command(s):	Signal Job Step by Node Request
Client:   	srun or slurmctld (or possibly scancel)
Server: 	slurmd
Input:  	uid, signal no., job_id
		optional: step_id, task no.
Output: 	return code
Notes:  	Signal all steps and all tasks unless otherwise specified.
		This could be used to support gang scheduling
		Cancel all steps unless otherwise specified.


Command(s):	Signal Job Step by Cluster Request
Client:   	scancel user command, plus DPCS from API, any node in cluster
Server: 	slurmctld
Input:  	uid, signal no., job_id
		optional: step_id, task no.
Output: 	return code
Notes:  	Can only be run as user root or the user id for the job
		Cancel all steps unless otherwise specified.


Command(s):	Job Step Attach Request
Client:   	srun
Server: 	slurmd
Input:  	uid, job_id, step_id
Output: 	return code, error code, 
		stdout/err duplicated to srun stdout/err, signals propagated,
Notes:  	srun process ``attaches'' to a currently running job. This
		request is used for srun recovery, or by a user who wants
		to interactively reattach to a batch job.


Command(s):	Reconfigure (tell slurmctld to re-read configuration)
Client:   	scontrol administrator command, any node in cluster
Server: 	slurmctld
Input:  	user id (root), version
Output: 	error code, version
Notes:  	can only be run as user root


Command(s):	Register node
Client:   	slurmd daemon, any node in cluster
Server: 	slurmctld
Input:  	version, time stamp, processor count, memory size, temporary disk space
Output: 	none
Notes:  	Done when slurmd restarts


Command(s):	Report node status
Client:   	slurmctld or backup slurmctld
Server: 	slurmd or slurmctld (for backup check) daemon, any node in cluster
Input:  	none
Output: 	version, time stamp, processor count, memory size, temporary disk space
Notes:  	


Command(s):	Upload accounting information
Client:   	slurmctld
Server: 	slurmd daemon, any node in cluster
Input:  	none
Output: 	version, time stamp, collection of records with user id, memory use, 
		CPU time used, CPU time allocated, etc.
Notes:  	not needed for initial release


Command(s):	Get job id from process id
Client:   	DPCS API
Server: 	slurmd daemon on the same node as DPCS API is executed
Input:  	process id
Output: 	SLURM job id
Notes:  	Until SLURM accounting is fully funcational, DPCS needs help figuring 
		out what processes are associated with each job. All message traffic 
		within a node


Command(s):	Get job step infomration
Client:   	srun from user script, slurmd, any node in cluster
Server: 	slurmctld
Input:  	job id, uid, step id
Output: 	step id, elan context (opaque data structure)
Notes:  	needed to start parallel program


Command(s):	Modify node information
Client:   	scontrol
Server: 	slurmctld
Input:  	node name, node state, uid
Output: 	exit code
Notes:  	Only the node state can be changed (esp. to DRAINING, DOWN, IDLE).
		Only request from user root is accepted

Command(s):	Modify partition information
Client:   	scontrol
Server: 	slurmctld
Input:  	partition name, allowed groups, default, key, max time, max nodes, 
		node list, shared, state, uid
Output: 	exit code
Notes:  	Only request from user root is accepted.


Command(s):	Modify job information
Client:   	scontrol
Server: 	slurmctld
Input:  	job id, time limit, priority, uid (must be root or owner
Output: 	exit code
Notes:  	Only request from the job's owner and user root are accepted.


Command(s):	Run epilog
Client:   	slurmctld
Server: 	slurmd
Input:  	job_id
Output: 	error code
Notes:  	On termination of a job (not the job step), slurmctld tells
		slurmd to execute its epilog program (if any). 


Summary of interactions:
dpcs->slurmd		Get job id from process id

scancel->slurmctld	Signal Job Step by Cluster Request

scancel->slurmd		Signal Job Step by Node Request

scontrol->slurmctld	Reconfigure
			Modify job information
			Modify node information
			Modify partition information
			Get job/accounting/node/partition/job_step/build information

slurmctld->slurmctld	Report node status (backup to primary controller)
		
slurmctld->slurmd	Signal Job Step by Node Request
			Report node status
			Run epilog
			Run Job Step Request
			Upload accounting information
			Signal Job Step by Node Request

slurmd->slurmctld	Get job step information
			Register node

srun->slurmctld		Get job step infomration
			Job Step Attach Request
			Get Job Step Informationt
			Register Job Step
			Submit job
			Allocate job
			Claim job allocation

srun->slurmd		Kill Job Step Request
			Signal Job Step by Node Request
			Run Job Step Request

----TEMPLATE----
Command(s):	
Client:   	
Server: 	
Input:  	
Output: 	
Notes:  	
