GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.
Tip
You can use GNU Parallel on your Mac OSX laptop/desktop to run applications in parallel. Just install using either brew or Macports
sudo port install parallel
brew install parallel
Availability¶
| Software | Version | Dependent Toolchain | Module Load Command |
|---|---|---|---|
| parallel | 20230722 | foss/2022b | module load foss/2022b parallel/20230722 |
Usage¶
Usage
module load parallel/20200422
parallel echo ::: 1 2 3 ::: 4 5 6
Purpose¶
GNU Parallel is a tool for running a series of jobs, mostly serial jobs, in parallel. This tool is best suited for running a bunch of serial jobs that may not run for the same simulation time. For example, the most easiest way to run a series of serial jobs in parallel on n cpus within one submit script is as follows
./job_1 &
./job_2 &
...
./job_n &
wait
./job_(n+1) &
./job_(n+2) &
...
./job_2n &
wait
./job_(2n+1) &
./job_(2n+2) &
...
./job_3n &
wait
This works most efficiently when all jobs have the same or almost the same run time. If run times for jobs are unequal, then n jobs are run simultaneously and the cpus remain idle until all n jobs are completed before looping through the next n jobs. This will lead to idle time and inefficient consumption of cpu time.
GNU Parallel solves this issue by first launching n jobs. When one job completes, then the next job in sequence is started. This permits efficient use of cpu time by reducing the wait time and letting a number of small jobs to run while some cpus work on longer jobs.
parallel job_{1} ::: $(seq 1 3n)
Single Node example¶
Multi Node example¶
Related Links¶
- https://www.gnu.org/software/parallel/parallel_tutorial.html
- https://www.biostars.org/p/63816/
- http://www.shakthimaan.com/posts/2014/11/27/gnu-parallel/news.html
- https://www.msi.umn.edu/support/faq/how-can-i-use-gnu-parallel-run-lot-commands-parallel
- https://github.com/LangilleLab/microbiome_helper/wiki/Quick-Introduction-to-GNU-Parallel
- https://davetang.org/muse/2013/11/18/using-gnu-parallel/
- https://sites.google.com/a/stanford.edu/rcpedia/parallel-processing/gnu-parallel-examples
- http://phili.pe/posts/free-concurrency-with-gnu-parallel/
User Contributed Information¶
Please help us improve this page
Users are invited to contribute helpful information and corrections through our Github repository.