Slurm: Reference Guide

How to use and get the best out of slurm, slurm scripts and efficient use fo the cluster.

Slurm

Jobs on eRI are submitted in the form of a batch script containing the code you want to run and a header of information needed by our job scheduler Slurm.
For full documentation on slurm and it’s usage see the Slurm website

Creating a batch script

Create a new file and open it with eg nano myjob.sl, the following should be considered as required for a job to start.

#!/bin/bash -e
#SBATCH --job-name=SerialJob # job name (shows up in the queue)
#SBATCH --account=2024-mjb-sandbox # project to record usage against
#SBATCH --time=00:01:00      # Walltime (days-HH:MM:SS)
#SBATCH --mem=512MB          # Memory in MB or GB

pwd # Prints working directory

Copy in the above text and save and exit the text editor with 'ctrl + x'.

Note:#!/bin/bashis expected by Slurm
Note: if you are a member of multiple accounts/projects you should use the relevant project code for the work to apply fairshare correctly.

Submitting

Jobs are submitted to the scheduler using:

sbatch myjob.sl

You should receive an output Submitted batch job 123456

Job Queue

The currently queued jobs can be checked using

squeue

You can filter to just your jobs by adding the flag

You can check all jobs submitted using:

Interactive Jobs

You can create an interactive session on the compute nodes (CPUs, MEM, time) for testing code and resource usage. Rather than using the login node which can result in system slowdowna nd blockages

Job Efficiency

How did my job run, what resources we acctually used. The outcome of which is others users are potentially blocked and/or your priority lowers. seff <JOBID>

Low MEM efficiency example, 256GB requested for 3 days, but only used 25GB. 4 of these jobs would fill an entire node and use 128 of the 256 CPUs. If 30GB were requested, then 8 jobs could be run on the same node.

Additional Slurm Commands

A complete list of Slurm commands can be found here, or by entering man slurm into a terminal

batch	`sbatch submit.sl`	Submits the Slurm script submit.sl
squeue	`squeue`	Displays entire queue.
	`squeue --me`	Displays your queued jobs.
	`squeue -p compute`	Displays queued jobs on the compute partition.
sacct	`sacct`	Displays all the jobs run by you that day.
	`sacct -S 2024-01-01`	Displays all the jobs run by you since the 1st Jan 2024
	`sacct -j 123456`	Displays job 123456
scancel	`scancel 123456`	Cancels job 123456
scancel	`scancel --me`	Cancels all your jobs.
sshare	`sshare -U`	Shows the Fair Share scores for all projects of which you are a member.
sinfo	`sinfo`	Shows the current state of the Slurm partitions.

sbatch options

A complete list of sbatch options can be found here, or by running “man sbatch”

Options can be provided on the command line or in the batch file as an #SBATCH directive. The option name and value can be separated using an '=' sign e.g. #SBATCH --account=2024-mjb-sandbox or a space e.g. #SBATCH --account 2024-mjb-sandbox. But not both!

General options

--job-name	`#SBATCH --job-name=MyJob`	The name that will appear when using squeue or sacct
--account	`#SBATCH --account=2024-mjb-sandbox`	The account that usage will be recorded for.
--time	`#SBATCH --time=DD-HH:MM:SS`	Job max walltime
--mem	`#SBATCH --mem=512MB`	Memory required per node.
--partition	`#SBATCH --partition=compute`	Specified job partition
--output	`#SBATCH --output=%j_output.out`	Standard output file.
--mail-user	`#SBATCH --mail-user=matt.bixley@agresearch.co.nz`	Address to send mail notifications.
--mail-type	`#SBATCH --mail-type=ALL`	Will send a mail notification at `BEGIN END FAIL`
--mail-type	`#SBATCH --mail-type=TIME_LIMIT_80`	Will send message at 80% walltime
--no-requeue	`#SBATCH --no-requeue`	Will stop job being requeued in the case of node failure.

Parallel options

--nodes	`#SBATCH --nodes=2`	Will request tasks be run across 2 nodes.
--ntasks	`#SBATCH --ntasks=2`	Will start 2 MPI tasks.
--ntasks-per-node	`#SBATCH --ntasks-per-node=1`	Will start 1 task per requested node
--cpus-per-task	`#SBATCH --cpus-per-task=10`	Will request 10 logical CPUs per task. See Hyperthreading.
--mem-per-cpu	`#SBATCH --mem-per-cpu=512MB`	Memory Per logical CPU. `--mem` Should be used if shared memory job. See How do I request memory?.
--array	`#SBATCH --array=1-5`	Will submit job 5 times each with a different `$SLURM_ARRAY_TASK_ID` (1,2,3,4,5)
	`#SBATCH --array=0-20:5`	Will submit job 5 times each with a different `$SLURM_ARRAY_TASK_ID` (0,5,10,15,20)
	`#SBATCH --array=1-100%10`	Will submit 1 though to 100 jobs but no more than 10 at once.

Other

--qos	`#SBATCH --qos=debug`	Adding this line gives your job a very high priority. Limited to one job at a time, max 15 minutes.
--profile	`#SBATCH --profile=ALL`	Allows generation of a .h5 file containing job profile information. See Slurm Native Profiling.
--dependency	`#SBATCH --dependency=afterok:123456789`	Will only start after the job 123456789 has completed.
--hint	`#SBATCH --hint=nomultithread`	Disables hyperthreading, be aware that this will significantly change how your job is defined.

Tip
Many options have a short and long form e.g. #SBATCH --job-name=MyJob & #SBATCH -J=MyJob.

Tokens

These are predefined variables that can be used in sbatch directives such as the log file name.

`%x`	Job name
`%u`	User name.
`%j`	Job ID
`%a`	Job array Index

Environment variables

Common examples.

`$SLURM_JOB_ID`	Useful for naming output files that won't clash.
`$SLURM_JOB_NAME`	Name of the job.
`$SLURM_ARRAY_TASK_ID`	The current index of your array job.
`$SLURM_CPUS_PER_TASK`	Useful as an input for multi-threaded functions.
`$SLURM_NTASKS`	Useful as an input for MPI functions.
`$SLURM_SUBMIT_DIR`	Directory where `sbatch` was called.

Tip
In order to decrease the chance of a variable being misinterpreted you should use the syntax ${NAME_OF_VARIABLE} and define in strings if possible. e.g.

Job Output

When the job completes, or in some cases earlier, two files will be added to the directory in which you were working when you submitted the job:

slurm-[jobid].out containing standard output.

slurm-[jobid].err containing standard error.

Highlight important information in a panel like this one. To edit this panel's color or style, select one of the options in the menu.

AgResearch eResearch Infrastructure