GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.
Tip
You can use GNU Parallel on your Mac OSX laptop/desktop to run applications in parallel. Just install using either brew or Macports
1 2 | |
AvailabilityPermanent link to Availability
| Software | Version | Dependent Toolchain | Module Load Command |
|---|---|---|---|
| parallel | 20240722 | foss/2024a | module load foss/2024a parallel/20240722 |
UsagePermanent link to Usage
Usage
1 2 | |
PurposePermanent link to Purpose
GNU Parallel is a tool for running a series of jobs, mostly serial jobs, in parallel. This tool is best suited for running a bunch of serial jobs that may not run for the same simulation time. For example, the most easiest way to run a series of serial jobs in parallel on n cpus within one submit script is as follows
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
This works most efficiently when all jobs have the same or almost the same run time. If run times for jobs are unequal, then n jobs are run simultaneously and the cpus remain idle until all n jobs are completed before looping through the next n jobs. This will lead to idle time and inefficient consumption of cpu time.

GNU Parallel solves this issue by first launching n jobs. When one job completes, then the next job in sequence is started. This permits efficient use of cpu time by reducing the wait time and letting a number of small jobs to run while some cpus work on longer jobs.
1 | |

Single Node examplePermanent link to Single Node example
Multi Node examplePermanent link to Multi Node example
Related LinksPermanent link to Related Links
- https://www.gnu.org/software/parallel/parallel_tutorial.html
- https://www.biostars.org/p/63816/
- http://www.shakthimaan.com/posts/2014/11/27/gnu-parallel/news.html
- https://github.com/LangilleLab/microbiome_helper/wiki/Quick-Introduction-to-GNU-Parallel
- https://davetang.org/muse/2013/11/18/using-gnu-parallel/
- https://sites.google.com/a/stanford.edu/rcpedia/parallel-processing/gnu-parallel-examples
- http://phili.pe/posts/free-concurrency-with-gnu-parallel/
User Contributed InformationPermanent link to User Contributed Information
Please help us improve this page
Users are invited to contribute helpful information and corrections through our Github repository.