Slurm
SLURM is a job scheduler. Its role is to manage the queue and launch calculations when the requested resources are available.
The partitions
The different calculation nodes are grouped by partitions. When submitting a job, you must choose a partition.
The partitions available are as follows:
- std : Includes all available CPU nodes.
- gpu : Includes all available GPU nodes.
- myXXX : CPU/GPU partitions accessible to contributors (users who have funded nodes).
To list the partitions enter the command $ sinfo
[cal01@vm-support01 explor-support]$ sinfo
PARTITION AVAIL AVAIL_FEATURES NODES NODES(A/I/O/T) NODELIST
std up BROADWELL,OPA,INT 121 107/10/4/121 cna[02-17,19-39,41-64],cnb[01-08,10-12,14-52,54-55,57-64]
std up SKYLAKE,OPA,INTEL 48 31/17/0/48 cnc[01-48]
std up SKYLAKE,OPA,INTEL 12 3/9/0/12 cnc[53-64]
std up IVY,IB,INTEL,NOPR 12 8/4/0/12 cnd[01-12]
std up BROADWELL,INTEL,H 16 1/15/0/16 cne[01-16]
std up SKYLAKE,OPA,INTEL 8 1/7/0/8 cnf[01-08]
std up CASCADELAKE,OPA,I 12 2/10/0/12 cng[01-12]
std up CASCADELAKE,OPA,I 2 0/2/0/2 cnh[01-02]
std up CASCADELAKE,IB,IN 28 5/23/0/28 cni[01-24,29-32]
std up CASCADELAKE,IB,IN 4 3/1/0/4 cni[25-28]
std up EPYC3,IB,AMD,NOPR 64 47/17/0/64 cnj[01-64]
std up CASCADELAKE,IB,IN 8 0/8/0/8 cnk[01-08]
std up EPYC4,IB,AMD 2 0/2/0/2 cnl[02-03]
std up EPYC4,IB,AMD,NOPR 1 1/0/0/1 cnl01
gpu up BROADWELL,OPA,P10 6 6/0/0/6 gpb[01-06]
gpu up BROADWELL,OPA,GTX 4 3/1/0/4 gpc[01-04]
gpu up CASCADELAKE,OPA,T 3 3/0/0/3 gpd[01-03]
gpu up CASCADELAKE,IB,RT 2 1/1/0/2 gpe[01-02]
gpu up CASCADELAKE,L40,I 1 0/1/0/1 gpf01
(A/I/O/T): (allocated/idle/other/total)
List the jobs submitted
To list the jobs submitted enter the command $ squeue
[cal01@vm-support01 ~]$ squeue
JOBID PARTITION NAME USER ACCOUNT ST TIME NODES QOS NODELIST(REASON)
1047837 gpu CoFe-Nv xfb86 cit51 PD 0:00 1 qos_01_16-00 (Resources)
1048214 gpu Test gnl27 hwn86 R 13:31:04 1 qos_01_16-00 gpc01
1048329 gpu N_tim_ps rgz30 efv45 R 2:02:43 1 qos_01_16-00 gpb06
1047749 gpu pestis rgz30 efv45 R 2-04:52:27 1 qos_01_16-00 gpb01
1046378 gpu N_15dhz rgz30 efv45 R 2-09:37:47 1 qos_01_16-00 gpb04
1046377 gpu N_10dhz rgz30 efv45 R 2-09:38:59 1 qos_01_16-00 gpb03
1047836 gpu CoFe-Nv xfb86 cit51 R 1-10:25:41 1 qos_01_16-00 gpb05
1044080 gpu N_CBU_Z_ rgz30 efv45 R 3-09:17:28 1 qos_01_16-00 gpb02
1044099 gpu CoFe-Nv xfb86 cit51 R 3-08:25:57 1 qos_01_16-00 gpe02
1025352 std 0Ni_3N mdl69 gzc01 CG 1-20:08:41 1 qos_08_04-00 cnj08
1040704 std Fe2-TiO2 zbs34 iwj80 CG 4:56 1 qos_04_08-00 cnj48
1040703 std Fe2-TiO2 zbs34 iwj80 CG 5:18 1 qos_04_08-00 cnj39
1006501 std IrO2bulk zbs34 iwj80 CG 8-00:00:17 1 qos_04_08-00 cnj16
1048549 std VlemGril fnk38 zmc46 PD 0:00 1 qos_01_16-00 (Resources)
1048545 std mpi rhq14 bcc72 PD 0:00 4 qos_04_08-00 (Priority)
1048334 std 1100 uwp08 nwe89 PD 0:00 16 qos_16_04-00 (Priority)
1048421 std 800 uwp08 nwe89 PD 0:00 16 qos_16_04-00 (Priority)
in graphical mode, you can also use the command $ sview.
Submitting a job
To submit a job, run the command :
$ sbatch script.slurm
Before submitting a job you must first load the necessary modules.
Some examples of submission scripts :
Example: calculation of 3 days of single processor
#!/bin/bash
## Partition
#SBATCH -p hf
## Number of nodes
#SBATCH -N 1
## Job name
#SBATCH -J mono-proc
## Number of tasks requested
#SBATCH -n 1
#SBATCH -t 3-00:00:00
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your_email_address@univ-lorraine.fr
module purge
module load gcc/7.1.0
## Creating temporary directory in $SCRATCHDIR
WORKDIR=$SCRATCHDIR/job.$SLURM_JOB_ID.$USER
mkdir -p $WORKDIR
## Copying files to $WORKDIR
cp -rf input executable $WORKDIR
## Moving to $WORKDIR
cd $WORKDIR
## Calculation
srun executable < input
## Copying back files to submit directory
cp -rf * $SLURM_SUBMIT_DIR/.
Example: 8 days and 30 min of calculation with OpenMP on 8 cores
#!/bin/bash
#SBATCH -N 1
#SBATCH -p hf
#SBATCH -J openmp
## Number of cpu per task
#SBATCH -c 8
#SBATCH -t 8-00:30:00
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your_email_address@univ-lorraine.fr
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
module purge
module load intel/2018.0.128 mkl/2018.0.128
## Creating temporary directory in $SCRATCHDIR
WORKDIR=$SCRATCHDIR/job.$SLURM_JOB_ID.$USER
mkdir -p $WORKDIR
## Copying files to $WORKDIR
cp -rf input executable $WORKDIR
## Moving to $WORKDIR
cd $WORKDIR
## Calculation
srun executable < input
## Copying back files to submit directory
cp -rf * $SLURM_SUBMIT_DIR/.
Example: calculation of 15 days and 3 hours with MPI on 2 nodes, 32 cores each
#!/bin/bash
#SBATCH -N 2
#SBATCH -p std
#SBATCH -J mpi
## Number of tasks requested
#SBATCH -n 64
## Number of tasks requested per node
#SBATCH --ntasks-per-node=32
#SBATCH -t 15-03
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your_email_address@univ-lorraine.fr
module purge
module load intel/2018.0.128 intelmpi/2018.0.128 mkl/2018.0.128
## Creating temporary directory in $SCRATCHDIR
WORKDIR=$SCRATCHDIR/job.$SLURM_JOB_ID.$USER
mkdir -p $WORKDIR
## Copying files to $WORKDIR
cp -rf input executable $WORKDIR
## Moving to $WORKDIR
cd $WORKDIR
## Calculation
srun executable < input
## Copying back files to submit directory
cp -rf * $SLURM_SUBMIT_DIR/.
Example: Calculation of 30min OpenMP + MPI on 2x32 cores
#!/bin/bash
#SBATCH -N 2
#SBATCH -p std
#SBATCH -J openmp
#SBATCH --ntasks-per-node=1
#SBATCH -c 32
#SBATCH -t 30:00
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your_email_address@univ-lorraine.fr
module purge
module load intel/2018.0.128 intelmpi/2018.0.128 mkl/2018.0.128
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
## Creating temporary directory in $SCRATCHDIR
WORKDIR=$SCRATCHDIR/job.$SLURM_JOB_ID.$USER
mkdir -p $WORKDIR
## Copying files to $WORKDIR
cp -rf input executable $WORKDIR
## Moving to $WORKDIR
cd $WORKDIR
## Calculation
srun executable < input
## Copying back files to submit directory
cp -rf * $SLURM_SUBMIT_DIR/.
Example: Calculation of 30min MPI on 8 cores and two GPUs on a gpu node
#!/bin/bash
#SBATCH -N 1
#SBATCH -p k20
#SBATCH -J mpi+gpu
#SBATCH -n 8
#SBATCH --gres=gpu:2
#SBATCH -t 30:00
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your_email_address@univ-lorraine.fr
module purge
module load intel/2018.0.128 intelmpi/2018.0.128 mkl/2018.0.128 cuda/8.0
## Creating temporary directory in $SCRATCHDIR
WORKDIR=$SCRATCHDIR/job.$SLURM_JOB_ID.$USER
mkdir -p $WORKDIR
## Copying files to $WORKDIR
cp -rf input executable $WORKDIR
## Moving to $WORKDIR
cd $WORKDIR
## Calculation
srun executable < input
## Copying back files to submit directory
cp -rf * $SLURM_SUBMIT_DIR/.
If you want to use the mpirun command instead of srun with the intelmpi module it is necessary to add in your submission scripts the command: unset I_MPI_PMI_LIBRARY
Cancelling a job
To cancel a submitted job enter the command $ scancel #JOBID, where #JOBID is the job number.
Interactive reservation of a calculation node (for compilation/debugging)
We recommend compiling your source codes on the compute nodes in order to take advantage of all optimizations and avoid overloading the front end.
To do this, you can reserve a node and connect to it in interactive mode.
Ex : to request a node of the std partition for 1 hour enter :
[****@vm-gce17 ~]$ salloc -N1 -p std -t 1:00:00 srun --pty bash
****@cna01's password:
Last login: Fri Apr 21 21:10:16 2017 from vm-gce17.prod.explor
[****@cna01 ~]$
To exit the interactive mode, use the command $ exit :
[****@cna01 ~]$ exit
logout
Connection to cna01 closed.
salloc: Job allocation 5232 has been revoked.