Building OpenFOAM to run parallel jobs on a Cluster using IntelMPI libraries

In this post we will explain how to build and run OpenFOAM in parallel on a cluster where:

- you don’t have admin rights on the Cluster

- the MPI libraries available on the Cluster are the Intel MPI implementation

- you are not allow to build OpenFOAM directly on the cluster

 

First let’s check what is the linux flavour of the Cluster:

　　[ald@Cluster127 ~]$ lsb_release -a

　　LSB Version:    :core-4.1-amd64:core-4.1-noarch

　　Distributor ID: CentOS

　　Description:    CentOS Linux release 7.7.1908 (Core)

　　Release:        7.7.1908

　　Codename:       Core

Since we cannot build OpenFOAM directly on the cluster, we will create in VirtualBox a virtual machine (VM) using the same operating system of the cluster  CentOS Linux release 7.7.1908, so that we end up with binaries that are compatible with the cluster. If you are new to VirtualBox and VM creation, you can follow this tutorial.

Once CentOS virtual machine is up and running we can install all the libraries dependencies required to build OpenFOAM from source code, a detailed guide can be found here, below is the summary list of all the yum command to be launched on the VM machine, as root:

　　sudo su -

　　yum -y install dnf-plugins-core

　　yum -y install epel-release

　　yum install system-config-users

　　yum groups mark install "Development Tools"

　　yum groups mark convert "Development Tools"

　　yum groupinstall "Development Tools"

　　yum install openmpi openmpi-devel zlib-devel gstreamer-plugins-base-devel -y

　　yum install libXext-devel libGLU-devel libXt-devel libXrender-devel -y

　　yum install libXinerama-devel libpng-devel -y

　　yum install libXrandr-devel libXi-devel libXft-devel -y

　　yum install libjpeg-turbo-devel libXcursor-devel -y

　　yum install readline-devel ncurses-devel python python-devel -y

　　yum install cmake qt-devel qt-assistant -y

　　yum install mpfr-devel gmp-devel -y

　　yum install zlib-static -y

　　yum install glibc-static -y

　　yum install qt5-assistant.x86_64 -y

　　yum install gstreamer-plugins-base.x86_64 -y

　　yum install --enablerepo=epel qtwebkit-devel -y

Then we can proceed to install the intelMPI library. For our purposes the Runtime license includes everything we need to run Intel MPI-based applications. The license is free and permanent. Here are the installation instruction to run intel MPI on CentOS7:

Create the YUM (DNF) repo file in the /temp directory as a normal user:

　　cd ~

　　tee > /tmp/oneAPI.repo << EOF

　　[oneAPI]

　　name=Intel® oneAPI repository

　　baseurl=https://yum.repos.intel.com/oneapi

　　enabled=1

　　gpgcheck=1

　　repo_gpgcheck=1

　　gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB

　　EOF

Move the newly created oneAPI.repo file to the configuration directory /etc/yum.repos.d:

　　sudo mv /tmp/oneAPI.repo /etc/yum.repos.d

Since our goal is to run heavily parallelized jobs on a cluster let’s install the intel high performance computing kit:  

　　sudo yum install intel-hpckit

It will require around 2.8GB in total.

　　Complete!

　　[serena@valkyrie ~]$

Once the installation is completed, the packages

　　intel-oneapi-mpi-2021.5.0.x86_64 0:2021.5.0-495                                                                          

　　intel-oneapi-mpi-devel.x86_64 0:2021.5.0-495

will be ready for use. To test if the installation was successful we can perform the test suggested here. Initialize the oneAPI environment with the following command:

　　. /opt/intel/oneapi/setvars.sh

If everything goes smoothly this command should output the message “:: oneAPI environment initialized ::”. Once checked that the setvars.sh script is working, we can add the following line at the end of the “~/.bashrc” file:

　　. /opt/intel/oneapi/setvars.sh

in this way when a new terminal is opened, setvars.sh is triggered automatically.

Alternatively it is possible to encapsulate the command in an alias, by adding inside “~/.bashrc”:

　　# Intel MPI start up

　　alias intelMPI='. /opt/intel/oneapi/setvars.sh'

Using this option, when a new terminal is opened, it is sufficient to type “intelMPI” to get intel one api environment initialized.

Once oneAPI environment is initialized we are ready to start the api with the command:  

        oneapi-cli

After that, select Create a Project -> cpp -> and from the list choose for example “matrix_mul” -> then use the tab key to select  “create” . You should get in output the following message as confirmation that oneAPI is working.

 

 

We can do one more additional test in order to check that intelMPI is ready to roll.

Navigate to the intel MPI installation path  /opt/intel/oneapi/mpi/2021.5.0/bin

[serena@valkyrie bin]$ cd /opt/intel/oneapi/mpi/2021.5.0/bin

Inside this directory the mpirun and mpicc scripts are present:

[serena@valkyrie bin]$ ls

cpuinfo  hydra_pmi_proxy  IMB-MT  mpif77  mpigcc  mpiicpc  

hydra_bstrap_proxy  IMB-MPI1   mpigxx   mpiifort

hydra_nameserver  mpicc  mpiexec.hydra  mpifc  mpirun   

Let’s check if mpirun works smoothly by doing the following test  :

[serena@valkyrie bin]$ mpirun -np 4 echo "hello"

hello

hello

hello

hello

OK. It works, we are then ready to compile OpenFOAM. Before launching the Allwmake, we need to modify the etc/bashrc file, since we want to use the IntelMPI libraries instead of the default OpenMPI.

NOTE: in the following explanation we will suppose that you have added inside your ~/.bashrc the following 2 aliases:

　　# User specific aliases and functions

　　alias OF2112='. ~/OpenFOAM/OpenFOAM-v2112/etc/bashrc'

　　alias intelMPI='. /opt/intel/oneapi/setvars.sh'

Open a terminal and launch the following commands:

        source ~/.bashrc

　　intelMPI

　　OF2112

　　foam

　　vi etc/bashrc

and inside set the WM_MPLIB variable as follows 

“export WM_MPLIB=INTELMPI”

Moreover inside the ~/.bashrc add the path of the intel mpi libraries:

　　vi ~/.bashrc

Add the following lines:

　　# INTELMPI

　　export PATH=/opt/intel/oneapi/mpi/2021.5.0/bin:$PATH

Source again  ~/.bashrc, reload OpenFOAM environment variables and start the build by using the following commands:

　　source ~/.bashrc

　　OF2112

　　foam

　　# start the building

　　./Allwmake -j 2>&1 | tee log.Allwmake_INTEL_MP

By inspecting the log.Allwmake_INTEL_MP file you will see on the top that the OpenFOAM build is using INTELMPI:

　　Starting compile OpenFOAM-v2112 Allwmake

　　Gcc system compiler

　　linux64GccDPInt32Opt, with INTELMPI intelmpi

Once the building completes, at the end of log.Allwmake_INTEL_MP you will see the following lines confirming that the compiling process finished successfully:

========================================

Finished compile of visualization with OpenFOAM-v2112

  Gcc system compiler

  linux64GccDPInt32Opt, with INTELMPI intelmpi

========================================

  OpenFOAM-v2112

  Gcc system compiler

  linux64GccDPInt32Opt, with INTELMPI intelmpi

  api   = 2112

  patch = 0

  bin   = 312 entries

  lib   = 152 entries

========================================

Let’s test if the motorbike tutorial can now be run in parallel inside the CentOS7.7 virtual machine using the IntelMPI libraries:  

serena@valkyrie OpenFOAM-v2112]$ tut

[serena@valkyrie tutorials]$ cd incompressible/simpleFoam/motorBike/

[serena@valkyrie motorBike]$ ls

0.orig  Allclean  Allrun  constant  system

[serena@valkyrie motorBike]$ ./Allclean

Cleaning case /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

[serena@valkyrie motorBike]$ ./Allrun

Running surfaceFeatureExtract on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running blockMesh on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running decomposePar on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running snappyHexMesh (6 processes) on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running topoSet (6 processes) on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Restore 0/ from 0.orig/  [processor directories]

Running patchSummary (6 processes) on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running potentialFoam (6 processes) on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running checkMesh (6 processes) on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Running simpleFoam (6 processes) on /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

Once simpleFoam parallel execution completes, we can check the log.simpleFoam file:  

Build  : _14aeaf8dab-20211220 OPENFOAM=2112 version=v2112

Arch   : "LSB;label=32;scalar=64"

Exec   : simpleFoam -parallel -decomposeParDict system/decomposeParDict.6

Date   : Jan 12 2022

Time   : 19:00:45

Host   : valkyrie

PID    : 18763

I/O    : uncollated

Case   : /home/serena/OpenFOAM/OpenFOAM-v2112/tutorials/incompressible/simpleFoam/motorBike

nProcs : 6

Hosts  :

(

    (valkyrie 6)

)

Parallel execution went smoothly and the 500 directory with the results is present:

[serena@valkyrie motorBike]$ ls

0.orig    constant          log.patchSummary        log.simpleFoam            

500       log.blockMesh     log.potentialFoam       log.snappyHexMesh    

Allclean  log.checkMesh     log.reconstructPar      log.surfaceFeatureExtract

Allrun    log.decomposePar  log.reconstructParMesh  log.topoSet   system

We are now ready to deploy our Virtual Machine OpenFOAM build into the cluster.

Compress the build and send the tgz file into the ~/OpenFOAM of the cluster via scp:

foam

cd ..

tar -zcvf OpenFOAM-v2112_IntelMPI.tgz OpenFOAM-v2112/

tar -zcvf ThirdParty-v2112_IntelMPI.tgz ThirdParty-v2112/

scp -r *tgz ald@127.**.**.**:~/OpenFOAM

Log into the cluster and uncompress the tgz files:

cd ~/OpenFOAM

tar -zxvf OpenFOAM-v2112_IntelMPI.tgz

tar -zxvf ThirdParty-v2112_IntelMPI.tgz

Source OpenFOAM bashrc using the following command:

source ~/OpenFOAM/OpenFOAM-v2112/etc/bashrc

You will get the following error:

openfoam (etc/config.sh/mpi) => invalid INTELMPI directory

   => unset

Please set I_MPI_ROOT (or MPI_ROOT) correctly. 

since we need to set the I_MPI_ROOT variable to point to the location where the intelMPI is installed. This location is usually provided by the admin Cluster documentation, in our case the location was /nisProduction/impi/2020.7.271/bin64/impi/2018.4.274/bin64/

Therefore we need to set the I_MPI_ROOT variable by launching in terminal the following command:

echo $I_MPI_ROOT  

/nisProduction/impi/2020.7.271/bin64/impi/2018.4.274/bin64/

Now if we relaunch the same previous command to source OpenFOAM

[ald@Cluster127 ~]$ source /nishome/ald/OpenFOAM/OpenFOAM-v2112/etc/bashrc

[ald@Cluster127 ~]$

nice and clean, no more error messages. Now the OpenFOAM commands should be available in our Cluster terminal, let’s check for example the following command

blockMesh

The blockMesh command is now recognized but it is immediately followed by the error message:

blockMesh: error while loading shared libraries: libmpi.so.12: cannot open shared object file: No such file or directory

To avoid this error, we need to set the path of the libmpi.so.12 library so that OpenFOAM can load it. This library should be located in the same path where the Intel MPI library is installed, in our case “/nisProduction/impi/2020.7.271/bin64/”. To find its exact location inside the cluster let’s launch the following command:

[ald@Cluster127 ~]$ find /nisProduction/impi/2020.7.271/bin64/ -name "libmpi.so.12"  

and the command gives the exact location of libmpi.so.12:    

/nisProduction/impi/2020.7.271/bin64/compilers_and_libraries_2018.6.288/linux/mpi/intel64/lib/release/libmpi.so.12

Therefore we can set the library path using the following command:

[ald@Cluster127 ~]$ export LD_LIBRARY_PATH=/nisProduction/impi/2020.7.271/bin64/compilers_and_libraries_2018.6.288/linux/mpi/intel64/lib/release/:$LD_LIBRARY_PATH

And if we enter inside an OpenFOAM case, for example the cavity tutorial and retry the “blockMesh” command, no more error:

[ald@Cluster127 cavity]$ blockMesh

Build  : _14aeaf8dab-20211220 OPENFOAM=2112 version=v2112

Arch   : "LSB;label=32;scalar=64"

Exec   : blockMesh

Date   : Jan 14 2022

Time   : 04:29:04

Host   : Cluster127

PID    : 5335

I/O    : uncollated

Case   : /hpc-master/ald/project/cavity

nProcs : 1

trapFpe: Floating point exception trapping enabled (FOAM_SIGFPE).

fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 5, max

FileModificationPolls 20)

allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

Create time

Creating block mesh from "system/blockMeshDict"

[...]

----------------

Patches

----------------

 patch 0 (start: 760 size: 20) name: movingWall

 patch 1 (start: 780 size: 60) name: fixedWalls

 patch 2 (start: 840 size: 800) name: frontAndBack

End

 

Serial execution is now successful. As next step we need to test a parallel execution using the slurm queueing system of the cluster. For this we will create the script “Takeoff_OF2112_parallel” to be saved inside the OpenFOAM case directory. Inside this script we need to copy the commands we previously used to set I_MPI_ROOT variable and the location of the libmpi.so.12. Here are the full contents of Takeoff_OF2112_parallel:

  

#!/bin/bash

#

#SBATCH --job-name=UBIK

#SBATCH --output=slurm_output.%j.out

#SBATCH --error=slurm_error.%j.err

#SBATCH -D /hpc-master/ald/project/cavity

#SBATCH --get-user-env

#SBATCH --clusters=Cluster127

#SBATCH --ntasks=4

#SBATCH --nodes=1

#SBATCH --ntasks-per-core=1

#SBATCH --exclusive

#SBATCH --partition=vshort_c16m128

rm -rf log* processor*  slurm_*

export CPUS=4

#Load the environment

source /nishome/ald/OpenFOAM/OpenFOAM-v2112/etc/bashrc

# MPI Intel

export I_MPI_ROOT=/nisProduction/impi/2020.7.271/bin64/impi/2018.4.274/bin64/

export PATH=/nisProduction/impi/2020.7.271/bin64/impi/2018.4.274/bin64/:$PATH

export LD_LIBRARY_PATH=/nisProduction/impi/2020.7.271/bin64/compilers_and_libraries_2018.6.288/linux/mpi/intel64/lib/release/:$LD_LIBRARY_PATH

# Source tutorial run functions

. $WM_PROJECT_DIR/bin/tools/RunFunctions

blockMesh >> log.blockMesh.$SLURM_JOB_ID

decomposePar -force >> log.decomposePar.$SLURM_JOB_ID

mpirun -np $CPUS icoFoam -parallel >& log.icoFoam.$SLURM_JOB_ID

And we are finally ready to submit the job and running it in parallel using the slurm “sbatch” command:

[ald@Cluster127 cavity]$ sbatch Takeoff_OF2112_parallel 

Submitted batch job 2290243 on cluster Cluster127

Once the job completes, we can check if the parallel execution was successful:

[ald@Cluster127 cavity]$ ls

0                      log.decomposePar.2290243  processor1  system

constant               log.icoFoam.2290243       processor2  Takeoff_OF2112_parallel

log.blockMesh.2290243  processor0                processor3

[ald@Cluster127 cavity]$ vi log.icoFoam.2290243

Build  : _14aeaf8dab-20211220 OPENFOAM=2112 version=v2112

Arch   : "LSB;label=32;scalar=64"

Exec   : icoFoam -parallel

Date   : Jan 14 2022

Time   : 04:32:22

Host   : n050

PID    : 8536

I/O    : uncollated

Case   : /hpc-master/ald/project/cavity

nProcs : 4

Hosts  :

(

   (n050 4)

)

[...]

Time = 0.5

Courant Number mean: 0.222158 max: 0.852134

[...]

ExecutionTime = 0.24 s  ClockTime = 1 s

End

Finalising parallel run

Parallel execution can now be run without issues.

Conclusion: with the above procedure we have shown how to safely install the required OpenFOAM dependencies packages and then build OpenFOAM inside a virtual Machine using the same Operating System and MPI libraries of the target Cluster machine. Once the binaries are copied inside the cluster, we can run the job in parallel by using a script where we set the paths to correctly initialize the MPI environment variables of the cluster. In this way you can test your new OpenFOAM builds without the need of Cluster administrator privileges.

REFERENCES

- Rehat intelMPI installation instruction

https://www.intel.com/content/www/us/en/develop/documentation/installation-guide-for-intel-oneapi-toolkits-linux/top/installation/install-using-package-managers/yum-dnf-zypper.html 

- Intel MPI installation instruction guide https://registrationcenter-download.intel.com/akdlm/irc_nas/1718/INSTALL.html?lang=en&fileExt=.html

- Getting started with intel one API

https://www.intel.com/content/www/us/en/develop/documentation/get-started-with-intel-oneapi-hpc-linux/top.html 

- Intel MPI Library for Redhat using the yum package manager

https://www.intel.com/content/www/us/en/developer/tools/oneapi/hpc-toolkit-download.html

- Intel Mpi library licencing

https://www.intel.com/content/www/us/en/developer/articles/troubleshooting/intel-mpi-library-licensing-faq.html 

 

 

お問い合わせ