Jobs that run on multiple nodes generally use a parallel programming API called MPI (Message Passing Interface), which allows processes on multiple nodes to communicate with high throughput and low latency (especially over Talapas' InfiniBand network).  MPI is a standard and has multiple implementations—several are available on Talapas, notably Open MPI and Intel MPI.  The choice between these is largely a matter of personal taste and the specific needs of the situation.

The SLURM scheduler has built-in support for MPI jobs.  Jobs can be run in a generic way, or if needed, you can use extra parameters to carefully control how MPI processes are mapped to the hardware.

General Principles

Most parts of job setup are the same for all MPI flavors.  Notably, you'll want to decide how many simultaneous tasks (processes) you want to run your job across.

Specifying Node and Task Counts

The most common method is to specify how many nodes you want the job to run on and how many tasks should be run on each node, see the example below. Also see sbatch script parameters for more information.

#SBATCH --partition=compute
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=28
#SBATCH --constraint=intel,e5-2690

This will result in an MPI job with 84 processes, 28 processes on each of the 3 nodes.  In this case, we've specified 28 tasks per node with the knowledge that our constraint of ‘intel,e5-2690’ will allocate only nodes which have 28 CPU cores to the job. 

See sinfo -o "%10R %8D %25N %8c %10m %40f %35G"for a complete list of nodes features relative to the partitions.

Although it's not necessary to use all cores on a node, this is often efficient, since more communication between processes happens on the same node.  That said, if the processes need more RAM than the default, you might need to run fewer tasks per node and specify a larger amount of memory per task.

Alternatively, you can simply specify the number of tasks and let SLURM place them on available nodes as it sees fit, for example:

#SBATCH --partition=compute
#SBATCH --ntasks=84
#SBATCH --constraint=amd,milan

The primary advantage of this approach is that the job will probably be scheduled sooner, since SLURM is free to use any available cores, rather than having to arrange for nodes with sufficient free cores to become available.  It’s recommended to keep the job tied to cores of the same type (though not required) through use of the --constraint flag.

Depending on the I/O properties of the job and number of nodes allocated, the job might run more slowly in this configuration, and runtime will vary a bit depending on exactly how the slots are spread across nodes.  If it works for your job, though, this could be a huge win, in terms of getting your job started sooner.

Whichever method you use, also consider the effect of job size on your wait time.  In particular, the more CPU cores you ask for, the longer you are likely to wait for your job to start.  For some jobs, there is a minimum CPU core count (because of the requirements of the software).  For others, the core count might be relatively arbitrary.  Usually adding more cores would be expected to make the job run more quickly.  Using fewer cores might lead to earlier job completion, though, if it results in your job starting significantly sooner.

Specifying Memory

For single-node jobs, it's common to use the SLURM --mem flag to specify the entire amount of memory the job will be allocated.  For multi-node jobs, though you will probably find it more intuitive and predictable to specify the amount of memory available to each individual task, like so

#SBATCH --mem-per-cpu=12G

This is strictly only needed if the job will require more than the default amount of RAM, but it's always a good idea.

Specifying SLURM Invocation

SLURM provides two slightly different ways to invoke your MPI program. 

1. The preferred way is to invoke it directly with the srun command within your sbatch script.  This provides a few additional features and is arguably a bit simpler.

2. The alternative is to invoke it using the mpirun/mpiexec program within your sbatch script. 

See the SLURM MPI guide for more information.

Intel MPI

To access the Intel OneAPI MPI compilers, such as, mpicc or mpiifort:

module load intel-oneapi-mpi/2021.9.0
mpicc helloworld_mpi.c -o helloworld_mpi.x

Next, create a batch script. For example, to use the recommended srun approach:

#!/bin/bash
#SBATCH --account=racs
#SBATCH --partition=compute
#SBATCH --job-name=intel-mpi
#SBATCH --output=intel-mpi.out
#SBATCH --error=intel-mpi.err
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=28
#SBATCH --ntasks-per-core=1
module load intel-oneapi-mpi/2021.9.0
srun ./helloworld_mpi.x

Open MPI

Starting an Open MPI job directly with srun is not supported.  Doing so might not produce an obvious error, but in some cases will simultaneously start many independent single-process jobs instead of a single MPI job with all processes working together.  At best this will be very slow, and at worst the output may be incorrect. For more information see man mpirun.

Pitfalls

Choosing parameters for MPI job submission can unfortunately be rather complicated.  One pitfall you may encounter is accidentally failing to make use of all requested CPU cores, leading to needlessly long job times and wasted resources.  To verify that all is well, check that you are getting significant speedups with increasing process count.  If your jobs don't run faster when you add cores, something is probably wrong.  You can also log into the compute nodes while your job is running to observe the processes and check that compute-bound processes are using 100% CPU; the htop command is useful here.

One combination that we've seen work quite poorly is specifying --nodes and --ntasks (with no --ntasks-per-node).  This seems to sometimes lead to the above wasted-resource problem.

If you have concerns, please reach out to us—we'd be happy to check for problems and make recommendations.

Related articles