\section{Related Work}
\subsection*{Portable Batch System (PBS)}

The Portable Batch System (PBS)~\cite{PBS}
is a flexible batch queuing and 
workload management system originally developed by Veridian Systems 
for NASA.  It operates on networked, multi-platform UNIX environments, 
including heterogeneous clusters of workstations, supercomputers, and 
massively parallel systems. PBS was developed as a replacement for 
NQS (Network Queuing System) by many of the same people.

PBS supports sophisticated scheduling logic (via the Maui 
Scheduler). 
PBS spawn's daemons on each 
machine to shepherd the job's tasks.
It provides an interface for administrators to easily 
interface their own scheduling modules.  PBS can support 
long delays in file staging with retry.  Host 
authentication is provided by checking port numbers (low ports numbers are only 
accessible to user root).  Credential service is used for user authentication. 
It has the job prolog and epilog feature.
PBS Supports 
high priority queue for smaller "interactive" jobs.  Signal to daemons 
causes current log file to be closed, renamed with 
time-stamp, and a new log file created.

Although the PBS is portable and has a broad user base, it has significant drawbacks.
PBS is single threaded and hence exhibits poor performance on large clusters. 
This is particularly problematic when a compute node in the system fails: 
PBS tries to contact down node while other activities must wait. 
PBS also has a weak mechanism for starting and cleaning up parallel jobs. 
%Specific complaints about PBS from members of the OSCAR group (Jeremy Enos, 
%Jeff Squyres, Tim Mattson):
%\begin{itemize}
%\item Sensitivity to hostname configuration on the server; improper 
%      configuration results in hard to diagnose failure modes.  Once 
%      configuration is correct, this issue disappears.
%\item When a compute node in the system dies, everything slows down.  
%      PBS is single-threaded and continues to try to contact down nodes,
%      while other activities like scheduling jobs, answering qsub/qstat 
%      requests, etc., have to wait for a complete timeout cycle before being
%      processed.
%\item Default scheduler is just FIFO, but Maui can be plugged in so this
%      is not a big issue.
%\item Weak mechanism for starting/cleaning up parallel jobs (pbsdsh).
%      When a job is killed, pbsdsh kills the processes it started, but
%      if the process doesn't die on the first shot it may continue on.
%\item PBS server continues to mark specific nodes offline, even though they 
%      are healthy.  Restarting the server fixes this.
%\item Lingering jobs.  Jobs assigned to nodes, and then bounced back to the 
%      queue for any reason, maintain their assignment to those nodes, even 
%      if another job had already started on them.  This is a poor clean up 
%      issue.
%\item When the PBS server process is restarted, it puts running jobs at risk.
%\item Poor diagnostic messages.  This problem can be as serious as ANY other 
%      problem.  This problem makes small, simple problems turn into huge 
%      turmoil occasionally.  For example, the variety of symptoms that arise 
%      from improper hostname configuration.  All the symptoms that result are 
%      very misleading to the real problem.
%\item Rumored to have problems when the number of jobs in the queues gets
%      large.
%\item Scalability problems on large systems.
%\item Non-portable to Windows
%\item Source code is a mess and difficult for others (e.g. the open source
%      community) to improve/expand.
%\item Licensing problems (see below).
%\end{itemize}
%The one strength mentioned is PBS's portability and broad user base.
%
%PBS is owned by Veridian and is released as three separate products with
%different licenses: {\em PBS Pro} is a commercial product sold by Veridian;
%{\em OpenPBS} is an pseudo open source version of PBS that requires 
%registration; and
%{\em PBS} is a GPL-like, true open source version of PBS.
%
%Bug fixes go into PBS Pro.  When a major revision of PBS Pro comes out,
%the previous version of PBS Pro becomes OpenPBS, and the previous version
%of OpenPBS becomes PBS.  The delay getting bug fixes (some reported by the
%open source community) into the true open source version of PBS is the source
%of some frustration.


\subsection{Quadrics RMS}

Quadrics RMS\cite{Quadrics02}
(Resource Management System) is for 
Unix systems having Quadrics Elan interconnects. 
RMS functionality and performance is excellent. 
It major limitation is the requirement for a Quadrics interconnect. 
The proprietary code and cost may also pose difficulties under some 
circumstances.


\subsection*{Maui Scheduler}

Maui Scheduler~\cite{Maui} is an advanced reservation HPC batch scheduler 
for use with SP, O2K, and UNIX/Linux clusters. 
It is widely used to extend the functionality of PBS and LoadLeveler, 
which Maui requires to perform the parallel job initiation and management.

\subsection*{Distributed Production Control System (DPCS)}

The Distributed Production Control System (DPCS)~\cite{DPCS}
is a scheduler developed at Lawrence Livermore National Laboratory (LLNL). 
The DPCS provides basic data collection and reporting
mechanisms for prject-level, near real-time accounting and resource allocation
to customers with established limits per customers' organization budgets,
In addition, the DPCS evenly distributes workload across available computers
and supports dynamic reconfiguration and graceful degradation of service to prevent
overuse of a computer where not authorized.
%DPCS is (or will soon be) open source, although its use is presently 
%confined to LLNL. The development of DPCS began in 1990 and it has 
%evolved into a highly scalable and fault-tolerant meta-scheduler 
%operating on top of LoadLeveler, RMS, and NQS. DPCS provides: 
%\begin{itemize}
%\item Basic data collection and reporting mechanisms for project-level, 
%      near real-time accounting.
%\item Resource allocation to customers with established limits per 
%      customers' organizational budgets. 
%\item Proactive delivery of services to organizations that are relatively 
%      underserviced using a fair-share resource allocation scheme.
%\item Automated, highly flexible system with feedback for proactive delivery 
%      of resources.
%\item Even distribution of the workload across available computers.
%\item Flexible prioritization of production workload, including "run on demand."
%\item Dynamic reconfiguration and re-tuning.
%\item Graceful degradation in service to prevent overuse of a computer where 
%      not authorized.
%\end{itemize}

DPCS supports only a 
limited number of computer systems: IBM RS/6000 and SP, Linux, 
Sun Solaris, and Compaq Alpha. 
Like the Maui Scheduler, DPCS requires an underlying infrastructure for 
parallel job initiation and management (LoadLeveler, NQS, RMS or SLURM).

\subsection*{LoadLeveler}

LoadLeveler~\cite{LoadLevelerManual,LoadLevelerWeb} 
is a proprietary batch system and parallel job manager by 
IBM. LoadLeveler supports few non-IBM systems. Very primitive 
scheduling software exists and other software is required for reasonable 
performance, such as Maui Scheduler or DPCS. 
The LoadLeveler has a  simple and very flexible queue and job class structure available 
operating in "matrix" fashion. 
The biggest problem of the LoadLeveler is its poor scalability. 
It typically requires 20 minutes to execute even a trivial 500-node, 8000-task
on the IBM SP computers at LLNL.
%In addition, all jobs must be initiated through the LoadLeveler, and a special version of
%MPI is requested to run a parallel job. 
%[So do RMS, SLURM, etc. for interconnect set-up - Moe]%
%
%Many configuration files exist with signals to 
%daemons used to update configuration (like LSF, good). All jobs must 
%be initiated through LoadLeveler (no real "interactive" jobs, just 
%high priority queue for smaller jobs). Job accounting is only available 
%on termination (very bad for long-running jobs). Good status 
%information on nodes and LoadLeveler daemons is available. LoadLeveler 
%allocates jobs either entire nodes or shared nodes ,depending upon configuration.
%
%A special version of MPI is required. LoadLeveler allocates 
%interconnect resources, spawns the user's processes, and manages the 
%job afterwards. Daemons also monitor the switch and node health using 
%a "heart-beat monitor." One fundamental problem is that when the 
%"Central Manager" restarts, it forgets about all nodes and jobs. They 
%appear in the database only after checking in via the heartbeat. It 
%needs to periodically write state to disk instead of doing 
%"cold-starts" after the daemon fails, which is rare. It has the job 
%prolog and epilog feature, which permits us to enable/disable logins 
%and remove stray processes.
%
%LoadLeveler evolved from Condor, or what was Condor a decade ago. 
%While I am less familiar with LSF and Condor than LoadLeveler, they 
%all appear very similar with LSF having the far more sophisticated 
%scheduler. We should carefully review their data structures and 
%daemons before designing our own.
%
\subsection*{Load Sharing Facility (LSF)}

LSF~\cite{LSF}
is a proprietary batch system and parallel job manager by 
Platform Computing. Widely deployed on a wide variety of computer 
architectures, it has sophisticated scheduling software including 
fair-share, backfill, consumable resources, an job preemption and 
very flexible queue structure.
It also provides good status information on nodes and LSF daemons.
While LSF is quite powerful, it is not open-source and can be costly on 
larger clusters.
%The LSF share many of its shortcomings with the LoadLeveler: job initiation only
%through LSF, requirement of a spwcial MPI library, etc.
%Limits are available on both a per process bs per-job  
%basis. Time limits include CPU time and wall-clock time. Many 
%configuration files with signals to daemons used to update 
%configuration (like LoadLeveler, good). All jobs must be initiated 
%through LSF to be accounted for and managed by LSF ("interactive" 
%jobs can be executed through a high priority queue for 
%smaller jobs). Job accounting only available in near real-time (important 
%for long-running jobs). Jobs initiated from same directory as 
%submitted from (not good for computer centers with diverse systems 
%under LSF control). Good status information on nodes and LSF daemons. 
%Allocates jobs either entire nodes or shared nodes depending upon 
%configuration.
%
%A special version of MPI is required. LSF allocates interconnect 
%resources, spawns the user's processes, and manages the job 
%afterwards. While I am less familiar with LSF than LoadLeveler, they 
%appear very similar with LSF having the far more sophisticated 
%scheduler. We should carefully review their data structures and 
%daemons before designing our own.


\subsection*{Condor}

Condor~\cite{Condor,Litzkow88,Basney97}
is a batch system and parallel job manager 
developed by the University of Wisconsin. 
Condor was the basis for IBM's LoadLeveler and both share very similar 
underlying infrastructure. Condor has a very sophisticated checkpoint/restart 
service that does not rely upon kernel changes, but a variety of 
library changes (which prevent it from being completely general). The 
Condor checkpoint/restart service has been integrated into LSF, 
Codine, and DPCS. Condor is designed to operate across a 
heterogeneous environment, mostly to harness the compute resources of 
workstations and PCs. It has an interesting "advertising" service. 
Servers advertise their available resources and consumers advertise 
their requirements for a broker to perform matches. The checkpoint 
mechanism is used to relocate work on demand (when the "owner" of a 
desktop machine wants to resume work).

%
%\subsection*{Linux PAGG Process Aggregates}
%
%PAGG~\cite{PAGG}
%consists of modifications to the linux kernel that allows
%developers to implement Process AGGregates as loadable kernel modules.
%A process aggregate is defined as a collection of processes that are
%all members of the same set. A set would be implemented as a container
%for the member processes. For instance, process sessions and groups
%could have been implemented as process aggregates.
%

\subsection*{Beowulf Distributed Process Space (BPROC)}

The Beowulf Distributed Process Space 
(BPROC)
is set of kernel
modifications, utilities and libraries which allow a user to start
processes on other machines in a Beowulf-style cluster~\cite{BProc}.  Remote
processes started with this mechanism appear in the process table
of the front end machine in a cluster. This allows remote process
management using the normal UNIX process control facilities. Signals
are transparently forwarded to remote processes and exit status is
received using the usual wait() mechanisms. This tight coupling of 
a cluster's nodes is convenient, but high scalability can be difficult 
to achieve.

%\subsection{xcat}
%
%Presumably IBM's suite of cluster management software 
%(xcat\footnote{http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/sg246041.html})
%includes a batch system.  Look into this.
%
%\subsection{CPLANT}
%
%CPLANT\footnote{http://www.cs.sandia.gov/cplant/} includes
%Parallel Job Launcher, Compute Node Daemon Process,
%Compute Node Allocator, Compute Node Status Tool.
%
%\subsection{NQS} 
%
%NQS\footnote{http://umbc7.umbc.edu/nqs/nqsmain.html}, 
%the Network Queueing System, is a serial batch system.
%
%\subsection*{LAM / MPI}
%
%Local Area Multicomputer (LAM)~\cite{LAM}
%is an MPI programming environment and development system for heterogeneous 
%computers on a network. 
%With LAM, a dedicated cluster or an existing network
%computing infrastructure can act as one parallel computer solving
%one problem.  LAM features extensive debugging support in the
%application development cycle and peak performance for production
%applications. LAM features a full implementation of the MPI
%communication standard.
%
%\subsection{MPICH}
%
%MPICH\footnote{http://www-unix.mcs.anl.gov/mpi/mpich/}
%is a freely available, portable implementation of MPI,
%the Standard for message-passing libraries.
%
%\subsection{Sun Grid Engine}
%
%SGE\footnote{http://www.sun.com/gridware/} is now proprietary.
%
%
%\subsection{SCIDAC}
%
%The Scientific Discovery through Advanced Computing (SciDAC) 
%project\footnote{http://www.scidac.org/ScalableSystems}
%has a Resource Management and Accounting working group
%and a white paper\cite{Res2000}. Deployment of a system with 
%the required fault-tolerance and scalability is scheduled 
%for June 2006.
%
%\subsection{GNU Queue}
%
%GNU Queue\footnote{http://www.gnuqueue.org/home.html}.
%
%\subsection{Clubmask}
%Clubmask\footnote{http://clubmask.sourceforge.net} is based on bproc.
%Separate queueing system?
%
%\subsection{SQMX}
%Part of the SCE Project\footnote{http://www.opensce.org/}, 
%SQMX\footnote{http://www.beowulf.org/pipermail/beowulf-announce/2001-January/000086.html} is worth taking a look at.
