How to use and get the best out of slurm, slurm scripts and efficient use fo the cluster.
LINKS TO BE UPDATED STILL
Slurm
Jobs on eRI are submitted in the form of a batch script containing the code you want to run and a header of information needed by our job scheduler Slurm.
For full documentation on slurm and it’s usage see the Slurm website
Creating a batch script
Create a new file and open it with eg nano myjob.sl, the following should be considered as required for a job to start.
| Code Block |
|---|
#!/bin/bash -e
#SBATCH --job-name=SerialJob # job name (shows up in the queue)
#SBATCH --account=2024-mjb-sandbox # project to record usage against
#SBATCH --time=00:01:00 # Walltime (days-HH:MM:SS)
#SBATCH --mem=512MB # Memory in MB or GB
pwd # Prints working directory
|
Copy in the above text and save and exit the text editor with 'ctrl + x'.
Note:#!/bin/bashis expected by Slurm
Note: if you are a member of multiple accounts/projects you should add the line #SBATCH --account=<projectcode>use the relevant project code for the work to apply fairshare correctly.
Submitting
Jobs are submitted to the scheduler using:
...
You should receive an output Submitted batch job 1748836
sbatchcan take command line arguments similar to those used in the shell script through SBATCH pragmas
You can find more details on its use on the Slurm Documentation123456
Job Queue
The currently queued jobs can be checked using
...
| Code Block |
|---|
squeue -u <userid>@agresearch.co.nz squeue -u matt.bixley@agresearch.co.nz |
You can also filter to just your jobs using
| Code Block |
|---|
squeue --me |
You can find more details on its use on the Slurm DocumentationYou can check all jobs submitted by you in the past day using:
| Code Block |
|---|
sacct |
Or since a specified date using:
| Code Block |
|---|
#all your job today
sacct -S YYYY-MM-DD |
Each job will show as multiple lines, one line for the parent job and then additional lines for each job step.
Tips
sacct -X Only show parent processes.
...
#since a specified date sacct -X #Only show parent processes. sacct --state=PENDING/RUNNING/FAILED/CANCELLED/TIMEOUT # Filter jobs by |
...
state |
Interactive Jobs
You can create an interactive session on the compute nodes (CPUs, MEM, time) for testing code and resource usage. Rather than using the login node which can result in system slowdowna nd blockages
...
Job Efficiency
How did my job run, did I waste resourceswhat resources we acctually used. The outcome of which is others users are potentially blocked and/or your priority lowers. seff <JOBID>
...
A complete list of Slurm commands can be found here, or by entering man slurm into a terminal
sbatchbatch |
| Submits the Slurm script submit.sl |
squeue |
| Displays entire queue. |
| Displays your queued jobs. | |
| Displays queued jobs on the compute partition. | |
sacct |
| Displays all the jobs run by you that day. |
| Displays all the jobs run by you since the 1st Jan 2024 | |
| Displays job 123456 | |
scancel |
| Cancels job 123456 |
| Cancels all your jobs. | |
sshare |
| Shows the Fair Share scores for all projects of which you are a member. |
sinfo |
| Shows the current state of the Slurm partitions. |
...
...
sbatch options
A complete list of sbatch options can be found here, or by running “man sbatch”
Options can be provided on the command line or in the batch file as an #SBATCH directive. The option name and value can be separated using an '=' sign e.g. #SBATCH --account=nesi999992024-mjb-sandbox or a space e.g. #SBATCH --account nesi999992024-mjb-sandbox. But not both!
General options
...
\uD83D\uDCCB Related articles
| Filter by label (Content by label) | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|