Running jobs¶

GPU jobs¶

Interactive GPU jobs¶

GPU jobs run on the gpu partition.

You can request an interactive GPU job with the srun Slurm command.

# Request 1 GPU on 1 node (2 CPU cores are allocated per GPU by default)
srun --partition=gpu --gpus=1 --pty /bin/bash
srun --partition=gpu --gres=gpu:1 --pty /bin/bash # also works

# Exit from the compute node to request a new node
exit  # or hit ctrl+d

Batch GPU jobs¶

You can schedule a batch GPU job with the sbatch Slurm command, which will queue your job for execution on an available GPU node. As with CPU jobs, we suggest debugging your job interactively (using srun) before submitting a batch job. Here is an example sbatch script for a GPU job:

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --cpus-per-task=2  # 2 CPUs per GPU (default ratio)
#SBATCH --gpus=1
#SBATCH --time=10:00
#SBATCH --job-name=GPU_Example

# Check that the GPU is available
nvidia-smi

# Run your GPU-accelerated application
python --version  # Replace this with your actual GPU-enabled command

sleep 5

Once the script is written, you can submit it to the scheduler with the sbatch command. Upon success, sbatch will return the ID it has assigned to the job

$ <username>@vcf-login-0:~$ sbatch gpu_job.sh 
Submitted batch job 186269

Check the job¶

Once submitted, the job enters the queue in the PENDING state. When resources become available and the job has sufficient priority, an allocation is created for it and it moves to the RUNNING state. If the job completes correctly, it goes to the COMPLETED state, otherwise, its state is set to FAILED.

You'll be able to check the status of your job and follow its evolution with the squeue -u $USER command:

$ <username>@vcf-login-0:~$ squeue -u $USER
     JOBID PARTITION     NAME          USER        ST   TIME    NODES  NODELIST(REASON)
      123       gpu      GPU_Example   <username>  R    0:12    1      vcf-node-0

The scheduler will automatically create an output file that will contain the result of the commands run in the script file. That output file is named slurm-.out by default, but can be customized via submission options. In the above example, you can list the contents of that output file with the following commands:

$ <username>@vcf-login-0:~$ cat slurm-186269.out
<nvidia-smi output showing the GPU(s) assigned to the job>
Python 3.12.2