mpi

Running MPI

• Setup
• Compiling the code
• Running MPI locally
• Running MPI on SBATCH

Setup

Make sure you can connect to the discovery cluster by using an ssh client:

ssh -X <usrid>@discovery.neu.edu

If error messages persist you can connect directly to one of the login nodes:

ssh -X <usrid>@discovery2.neu.edu or
ssh -X <usrid>@discovery4.neu.edu

Make sure the following modules are loaded in your bashrc file:

vim ~/.bashrc

Edit to add these lines at the end of the file:

module load slurm-14.11.8 
module load gnu-4.8.1-compilers 
module load fftw-3.3.3 
module load platform-mpi 
module load cuda-7.0 

Notice that we are loading the module platform-mpi. This is the one we will use to be able to execute and compile mpi code. Note: you will need to logout and login for these changes to take effect. Make sure that openmpi is not loaded in your .bashrc file. You can add the following lines as well to make it easier to use slurm:

alias sq="squeue -u $USER"
alias sc="scancel"
alias sb="sbatch"
alias sa="salloc -N 1 --exclusive"

Requesting a compute node for compilation and/or interactive MPI runs Check systems for availability by using the sinfo command: sinfo or sinfo | grep idle # To identify compute nodes that are available

Allocate an interactive node by using the following command. Only use the ser-par-10g partitions for MPI runs:

salloc -N 1 --exclusive -p <partition name>

And then use squeue to identify which compute node was allocated, and ssh into the compute node.

squeue $USER
ssh -X compute-<#>

Example:

Compiling the code

The example code can be found in the directory

/scratch/gutierrez.jul/HPC/MPI/

Please copy this folder into your home directory.

cp -r /scratch/gutierrez.jul/HPC/MPI ~/

Go inside this folder and look at the files that are there. Understand what the Makefile, bash and cpp files do before moving forward.

cd ~/MPI

Compile the code using the makefile by executing the following command:

make all

In this example we use mpiCC to compile C++ code.

Running MPI locally

We can run MPI in the interactive node we are using to compile the code by using the mpirun command. The following is an example of the command and it’s execution:

mpirun -np 20 -prot -TCP ./hello_mpi

An option to run this same command has been added to the Makefile. This can be launched by running:

make run

Running MPI on SBATCH

For this example, we will use the script named submit.bash. Read and understand what this script is doing. Notice this script will use the run.sbatch script as well. Execute it by running the following command and following the prompt:

./submit.bash

Once the job has completed, it will generate 2 files, a <job_name>.out and a <job_name>.err. Check that the file .err is completely empty. If there are errors, try to fix them. If not, check for the output on the .out file. Example of this execution run is as follows:

Now that you have tried it out, implement your own code using this guideline and hopefully everything should work fine. Feel free to modify the scripts to suite your needs.