Managing and Monitoring Jobs
Overview¶
Managing and monitoring your jobs effectively helps ensure efficient use of resources and enables quicker debugging when things go wrong. Slurm provides several built-in commands to track job status, usage, and troubleshoot issues.
SLURM has numerous tools for monitoring jobs. Below are a few to get started. More documentation is available on the SLURM website.
The most common commands are:
- List all current jobs:
squeue - Job deletion:
scancel [job_id] - Run a job:
sbatch [submit script] - Run a command:
srun <slurm options> <command name>
SLURM User Commands¶
| Task | Command |
|---|---|
| Job submission: | sbatch [script_file] |
| Job deletion: | scancel [job_id] |
| Job status by job: | squeue [job_id] |
| Job status by user: | squeue -u [user_name] |
| Job hold: | scontrol hold [job_id] |
| Job release: | scontrol release [job_id] |
| List enqueued jobs: | squeue |
| List nodes: | sinfo -N OR scontrol show nodes |
| Cluster status: | sinfo |
Use these commands to manage and monitor your jobs¶
- The
reportseffcommand is a handy tool to assess how efficiently your job used the requested resources after it has completed. - Using
reportseffcan help you adjust your future job scripts to request only as much memory/time as truly needed, improving scheduler fairness and reducing wasted resources.
Pass the ID of a finished job to reportseff:
Example output
Each efficiency column compares what the job actually used with what it requested:
| Column | How it is calculated | What it tells you |
|---|---|---|
TimeEff |
Elapsed run time ÷ requested walltime (--time) |
How much of the requested time the job needed. A low value means --time can be shortened, which often lets the job start sooner. |
CPUEff |
CPU time used ÷ (allocated cores × elapsed time) | How busy the allocated cores were. A low value means cores sat idle, so request fewer, or check that the code actually runs in parallel. |
MemEff |
Peak memory used ÷ requested memory | How much of the requested memory the job used. A low value means the memory request can be lowered. |
In the example above, the job finished in 28 seconds and used only 0.4% of its requested time and 0.1% of its requested memory, so both requests were far larger than needed. Its cores were busy 67.9% of the time.
- The
squeuecommand lets you check all the jobs currently running/pending/queued in the Wulver. - You can use
squeue -u $LOGNAMEto check your your jobs in the queue.
Example Output
- The
scancelcommand lets you cancel your job. - It take your JobID as argument
scancel [job_id]
- The
quota_infocommand lets you check your storage space and SUs consumed. More info
Example Output
The checkq command gives same output as squeue with extra details.
Example Output
The checkload command gives you the cpu load on each node.
Example Output
The slurm_jobid [job_id] command lets you check detailed info about your job. It requires your job_id as parameter.
Example Output
Info
Please keep checking your job's status using squeue -u $LOGNAME so that it doesn't stay pending due to incorrect job submission parameters.
Warning
You can only cancel your jobs. Please don't try to cancel other users jobs.