Page Comparison

How to use and get the best out of slurm, slurm scripts and efficient use fo the cluster

100% copy and paste from NeSI Slurm guide at the moment, links to NeSI slurm and resources may not provide the correct detail.

\uD83D\uDCD8 Instructions

If you are unsure about using our job scheduler Slurm, more details can be found here.

.

Slurm

Jobs on eRI are submitted in the form of a batch script containing the code you want to run and a header of information needed by our job scheduler Slurm.
For full documentation on slurm and it’s usage see the Slurm website

Creating a batch script

Create a new file and open it with eg nano myjob.sl, the following should be considered as required for a job to start.

#!/bin/bash -e
#SBATCH --job-name=SerialJob # job name (shows up in the queue)
#SBATCH --account=2024-mjb-sandbox # project to record usage against
#SBATCH --time=00:01:00      # Walltime (days-HH:MM:SS)
#SBATCH --mem=512MB          # Memory in MB or GB

pwd # Prints working directory

Copy in the above text and save and exit the text editor with 'ctrl + x'.

Note:#!/bin/bashis expected by Slurm
Note: if you are a member of multiple accounts/projects you should use the relevant project code for the work to apply fairshare correctly.

Submitting

Jobs are submitted to the scheduler using:

sbatch myjob.sl

You should receive an output Submitted batch job 123456

Job Queue

The currently queued jobs can be checked using 

squeue

You can filter to just your jobs by adding the flag

squeue -u <userid>@agresearch.co.nz
squeue -u matt.bixley@agresearch.co.nz
squeue --me

You can check all jobs submitted using:

sacct               #all your job today
sacct -S YYYY-MM-DD #since a specified date
sacct -X            #Only show parent processes.
sacct --state=PENDING/RUNNING/FAILED/CANCELLED/TIMEOUT # Filter jobs by state

Interactive Jobs

You can create an interactive session on the compute nodes (CPUs, MEM, time) for testing code and resource usage. Rather than using the login node which can result in system slowdowna nd blockages

srun --cpus-per-task 2 --account 2024-mjb-sandbox --mem 6G -p compute --time 01:00:00 --pty bash

Job Efficiency

How did my job run, what resources we acctually used. The outcome of which is others users are potentially blocked and/or your priority lowers. seff <JOBID>

Low MEM efficiency example, 256GB requested for 3 days, but only used 25GB. 4 of these jobs would fill an entire node and use 128 of the 256 CPUs. If 30GB were requested, then 8 jobs could be run on the same node.

login-0 ~ $ seff 391751_28
Job ID: 394314
Array Job ID: 391751_28
Cluster: eri
User/Group: bixleym@agresearch.co.nz/bixleym@agresearch.co.nz
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 32
CPU Utilized: 79-07:10:55
CPU Efficiency: 76.80% of 103-06:03:12 core-walltime
Job Wall-clock time: 3-05:26:21
Memory Utilized: 25.34 GB
Memory Efficiency: 9.90% of 256.00 GB

Additional Slurm Commands

A complete list of Slurm commands can be found here, or by entering man slurm into a terminal

...

...

sbatch options

A complete list of sbatch options can be found here, or by running “man sbatch”

Options can be provided on the command line or in the batch file as an #SBATCH directive.  The option name and value can be separated using an '=' sign e.g. #SBATCH --account=nesi999992024-mjb-sandbox or a space e.g. #SBATCH --account nesi999992024-mjb-sandbox. But not both!

General options

...

Tip

In order to decrease the chance of a variable being misinterpreted you should use the syntax ${NAME_OF_VARIABLE} and define in strings if possible. e.g.

1. Code Block
   echo "Completed task ${SLURM_ARRAY_TASK_ID} / ${SLURM_ARRAY_TASK_COUNT} successfully"

Slurm

Jobs on eRI are submitted in the form of a batch script containing the code you want to run and a header of information needed by our job scheduler Slurm.

Creating a batch script

Create a new file and open it with nano myjob.sl, the following should be considered as required for a job to start.

#!/bin/bash -e
#SBATCH --job-name=SerialJob # job name (shows up in the queue)
#SBATCH --account=2024-mjb-sandbox # project to record usage against
#SBATCH --time=00:01:00      # Walltime (days-HH:MM:SS)
#SBATCH --mem=512MB          # Memory in MB or GB

pwd # Prints working directory

Copy in the above text and save and exit the text editor with 'ctrl + x'.

Note:#!/bin/bashis expected by Slurm

Note: if you are a member of multiple accounts you should add the line #SBATCH --account=<projectcode>

Submitting

Jobs are submitted to the scheduler using:

sbatch myjob.sl

You should receive an output

Submitted batch job 1748836

sbatchcan take command line arguments similar to those used in the shell script through SBATCH pragmas

You can find more details on its use on the Slurm Documentation

Job Queue

The currently queued jobs can be checked using 

squeue

You can filter to just your jobs by adding the flag

squeue -u <userid>@agresearch.co.nz
squeue -u matt.bixley@agresearch.co.nz

You can also filter to just your jobs using

squeue --me

You can find more details on its use on the Slurm Documentation

You can check all jobs submitted by you in the past day using:

sacct

Or since a specified date using:

sacct -S YYYY-MM-DD

Each job will show as multiple lines, one line for the parent job and then additional lines for each job step.

Tips

sacct -X Only show parent processes.

sacct --state=PENDING/RUNNING/FAILED/CANCELLED/TIMEOUT Filter jobs by state.

You can find more details on its use on the Slurm Documentation

...

scancel <jobid> will cancel the job described by <jobid>. You can obtain the job ID by using sacct or squeue.

Tips

scancel -u [username] Kill all jobs submitted by you.

scancel {[n1]..[n2]} Kill all jobs with an id between [n1] and [n2]

...

Job Output

When the job completes, or in some cases earlier, two files will be added to the directory in which you were working when you submitted the job:

...

\uD83D\uDCCB Related articles