Difference between revisions of "Using the queue system on skd-cyclone"

From gfi
(Created page with "General information - Purpose - Advantages - Disadvantages - Rules and procedures Short tutorial - copy a basic script to your home - submit the script - receive an email...")
 
 
(33 intermediate revisions by 2 users not shown)
Line 1: Line 1:
General information
+
== General information ==
  
- Purpose
+
===Purpose===
- Advantages
 
- Disadvantages
 
- Rules and procedures
 
  
Short tutorial
+
The machine skd-cyclone.klientdrift.uib.no is used as a general-purpose linux computer with several connected storage. Researchers, PostDocs, PhD students and Master students use the machine for calculations. On several occasions, this has lead shortages in calculation resources, concerning both memory and CPU. The machine has started to "swap" memory to the disk, slowing down all calculations on the computer to an extend where it can not be used any more.
  
- copy a basic script to your home
+
The queuing system on skd-cyclone is in place to schedule jobs according to the available resources on the machine. This leads to a more balanced load of the machine, and avoids swapping situations.
- submit the script
 
- receive an email
 
  
Converting a script to a queue script
+
===Advantages===
  
- add elements to the header
+
The main advantage of the queue system is that is makes calculations on skd-cyclone more stable and reliable, and thus overall faster.
  
Description of each element
+
Further advantages are:
  
- parameters explained in short
+
* Interactive run of programs still possible
 +
* Lightweight wrapper scripts required to run jobs on queue
 +
* Email on fail/completion of job
 +
* Queues with different priority
 +
* The queue at GFI is a way of introducing already MSc students to requirements of a HPC environment
  
Best practices, Q&A
+
===Disadvantages===
  
Further information
+
* No use of screen or similar commands to interactively connect to running jobs
 +
* Waiting time when queue is full
 +
* Need to write "wrapper script" (see below)
  
- Contacts
+
===Rules and procedures===
- Links
+
 
 +
* The queue system is opt-in basis. That means that there is no obligation to use the queue system. Small jobs, for example, can run directly on the machine. Users who repeatedly run large jobs, however, such as NWP models or diagnostic code, will be asked to submit their job to the queue system.
 +
 
 +
== Short tutorial ==
 +
 
 +
1. create a basic wrapper script for your job
 +
 
 +
<pre>
 +
  #!/bin/bash
 +
  #
 +
  # SLURM wrapper script
 +
  # start this job with sbatch NAME_OF_THIS_FILE directly on the machine
 +
  #
 +
  #SBATCH --job-name=my_slurm_job
 +
  #SBATCH --workdir=/scratch/$USER
 +
  #SBATCH --partition=normal
 +
  #SBATCH --output=results.%j.txt
 +
  #SBATCH --error=errors.%j.out
 +
  #SBATCH --mail-type=END
 +
  #SBATCH --mail-type=FAIL
 +
  #SBATCH --ntasks=1
 +
  #SBATCH --cpus-per-task=4
 +
 
 +
  #apply any required path settings
 +
  export PATH=$PATH:/Data/gfi/met/bin
 +
 
 +
  #change to your working directory
 +
  cd /Data/gfi/met/batch
 +
 
 +
  # command to start your job
 +
  ./run_flexpart_noresm.sh
 +
 
 +
  # done
 +
  </pre>
 +
 
 +
2. submit the script to the queue
 +
 
 +
<code>sbatch ./my_queue_job.sh</code>
 +
 
 +
3. monitor your jobs and the general load on the queue
 +
 
 +
<code>squeue</code>
 +
 
 +
4. examine what resources are used by your job
 +
 
 +
<code>sstat <jobid></code>
 +
 
 +
5. cancel a job if required
 +
 
 +
<code>scancel <jobid></code>
 +
 
 +
6. receive an email when the job is finished
 +
 
 +
For example with the following header:
 +
 
 +
<pre>
 +
From: slurm@klientdrift.uib.no
 +
To: <username>@uib.no
 +
Subject: SLURM Job_id=34 Name=WaterSip3.0 Ended, Run time 00:01:00
 +
</pre>
 +
 
 +
== Converting a script to a queue script ==
 +
 
 +
# add elements to the header
 +
 
 +
== Description of each element ==
 +
 
 +
The SLURM queue is controlled by a set of parameters at the beginning of a shell script:
 +
#SBATCH --<parameter>=<argument>
 +
These parameters control for example the output directory, emails, resource allocation and emailing.
 +
 
 +
'''The most important aspect is to determine the memory demands of your job.''' This is nicely described here:
 +
[https://rc.fas.harvard.edu/resources/documentation/slurm-memory/ Finding out which memory limits to use]
 +
 
 +
The most common script parameters are explained here in short:
 +
 
 +
*'''mem''': how much memory your job will use at most (in MB).
 +
Example: <code> --mem 1000</code>
 +
*'''workdir''': output files will be created in this directory.
 +
Example: <code> --workdir=/scratch/$USER</code>
 +
*'''queue''': which queue the output should be processed on.
 +
Example: <code>--partition=default</code>
 +
*'''job-name''': provide a meaningful name to your job that will be visible on the queue
 +
Example: <code>--job-name=my_slurm_job</code>
 +
*'''output''': normal output will be written to this file. %j is a placeholder for the jobid
 +
Example: <code>--output=results.%j.txt</code>
 +
*'''error''': error messages will be written to this file. %j is a placeholder for the jobid
 +
Example: <code>--error=errors.%j.out</code>
 +
*'''mail-type''': an email will be send on normal completion (parameter END) or error (parameter FAIL) or for all occations (parameter ALL). For a combination of parameters (e.g. END and FAIL) the line must be included several times.
 +
Example: <code>--mail-type=ALL</code>
 +
*'''time''': maximum requested computation time (12h in the example below)
 +
Example: <code>--time=12:00:00</code>
 +
*'''ntasks''': number of cores requested, for example for a MPI parallelisation
 +
Example:<code>--ntasks=1</code>
 +
*'''cpus-per-task''': number of parallel threads to be run, for example for OpenMP parallelisation
 +
Example: <code>--cpus-per-task=4</code>
 +
*'''mem-per-cpu''': requested memory per CPU (100MB in the example below)
 +
Example: <code>--mem-per-cpu=100</code>
 +
 
 +
== Best practices, Q&A ==
 +
 
 +
== Further information ==
 +
 
 +
==== Contacts ====
 +
 
 +
In case of questions, please contact Idar Hessevik or Harald Sodemann at GFI
 +
 
 +
==== Links ====
 +
 
 +
[http://tanviralamspace.blogspot.no/2015/01/slurm-tutorial-basic-commands.html A very readable blog about the SLURM queue]
 +
 
 +
[http://www.ceci-hpc.be/slurm_tutorial.html A easy-to-understand SLURM documentation]
 +
 
 +
[http://slurm.schedmd.com/ The full SLURM documentation]

Latest revision as of 11:56, 22 May 2017

General information

Purpose

The machine skd-cyclone.klientdrift.uib.no is used as a general-purpose linux computer with several connected storage. Researchers, PostDocs, PhD students and Master students use the machine for calculations. On several occasions, this has lead shortages in calculation resources, concerning both memory and CPU. The machine has started to "swap" memory to the disk, slowing down all calculations on the computer to an extend where it can not be used any more.

The queuing system on skd-cyclone is in place to schedule jobs according to the available resources on the machine. This leads to a more balanced load of the machine, and avoids swapping situations.

Advantages

The main advantage of the queue system is that is makes calculations on skd-cyclone more stable and reliable, and thus overall faster.

Further advantages are:

  • Interactive run of programs still possible
  • Lightweight wrapper scripts required to run jobs on queue
  • Email on fail/completion of job
  • Queues with different priority
  • The queue at GFI is a way of introducing already MSc students to requirements of a HPC environment

Disadvantages

  • No use of screen or similar commands to interactively connect to running jobs
  • Waiting time when queue is full
  • Need to write "wrapper script" (see below)

Rules and procedures

  • The queue system is opt-in basis. That means that there is no obligation to use the queue system. Small jobs, for example, can run directly on the machine. Users who repeatedly run large jobs, however, such as NWP models or diagnostic code, will be asked to submit their job to the queue system.

Short tutorial

1. create a basic wrapper script for your job

  #!/bin/bash
  #
  # SLURM wrapper script
  # start this job with sbatch NAME_OF_THIS_FILE directly on the machine
  #
  #SBATCH --job-name=my_slurm_job
  #SBATCH --workdir=/scratch/$USER
  #SBATCH --partition=normal
  #SBATCH --output=results.%j.txt
  #SBATCH --error=errors.%j.out
  #SBATCH --mail-type=END
  #SBATCH --mail-type=FAIL
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=4

  #apply any required path settings
  export PATH=$PATH:/Data/gfi/met/bin

  #change to your working directory
  cd /Data/gfi/met/batch

  # command to start your job
  ./run_flexpart_noresm.sh

  # done
  

2. submit the script to the queue

sbatch ./my_queue_job.sh

3. monitor your jobs and the general load on the queue

squeue

4. examine what resources are used by your job

sstat <jobid>

5. cancel a job if required

scancel <jobid>

6. receive an email when the job is finished

For example with the following header:

From: slurm@klientdrift.uib.no
To: <username>@uib.no
Subject: SLURM Job_id=34 Name=WaterSip3.0 Ended, Run time 00:01:00

Converting a script to a queue script

  1. add elements to the header

Description of each element

The SLURM queue is controlled by a set of parameters at the beginning of a shell script:

#SBATCH --<parameter>=<argument>

These parameters control for example the output directory, emails, resource allocation and emailing.

The most important aspect is to determine the memory demands of your job. This is nicely described here: Finding out which memory limits to use

The most common script parameters are explained here in short:

  • mem: how much memory your job will use at most (in MB).

Example: --mem 1000

  • workdir: output files will be created in this directory.

Example: --workdir=/scratch/$USER

  • queue: which queue the output should be processed on.

Example: --partition=default

  • job-name: provide a meaningful name to your job that will be visible on the queue

Example: --job-name=my_slurm_job

  • output: normal output will be written to this file. %j is a placeholder for the jobid

Example: --output=results.%j.txt

  • error: error messages will be written to this file. %j is a placeholder for the jobid

Example: --error=errors.%j.out

  • mail-type: an email will be send on normal completion (parameter END) or error (parameter FAIL) or for all occations (parameter ALL). For a combination of parameters (e.g. END and FAIL) the line must be included several times.

Example: --mail-type=ALL

  • time: maximum requested computation time (12h in the example below)

Example: --time=12:00:00

  • ntasks: number of cores requested, for example for a MPI parallelisation

Example:--ntasks=1

  • cpus-per-task: number of parallel threads to be run, for example for OpenMP parallelisation

Example: --cpus-per-task=4

  • mem-per-cpu: requested memory per CPU (100MB in the example below)

Example: --mem-per-cpu=100

Best practices, Q&A

Further information

Contacts

In case of questions, please contact Idar Hessevik or Harald Sodemann at GFI

Links

A very readable blog about the SLURM queue

A easy-to-understand SLURM documentation

The full SLURM documentation