OpenFOAM, which stands for Open Field Operation and Manipulation, is a free, open-source software framework primarily used for computational fluid dynamics (CFD) simulations. It provides a wide range of numerical methods and solvers to simulate fluid flow, heat transfer, chemical reactions, and other related phenomena.
Developed by the OpenFOAM Foundation, OpenFOAM is written in C++ and includes a comprehensive set of libraries and tools for simulating complex fluid flows in various engineering and scientific applications. It supports various mesh types, including structured, unstructured, and Cartesian meshes, and offers parallel computing capabilities to leverage multicore processors and high-performance computing clusters.
OpenFOAM is widely used in automotive, aerospace, energy, and environmental engineering industries for tasks like aerodynamics analysis, combustion modelling, multiphase flow simulations, and more. Its open-source nature allows users to customize and extend its functionality according to their specific needs, making it a popular choice for academic research and industrial applications.
Azure CycleCloud offers effective and efficient solutions for running OpenFOAM simulations, whether for individuals or organizations. In this particular scenario, we'll explore a Slurm-based high-performance computing (HPC) setup utilizing HBv4 virtual machines and Ubuntu 22.04.
The following table shows the components used in the demonstration.
|
CycleCloud |
|
|
Slurm |
23.02.6-1 |
|
OS |
|
|
OpenFOAM |
|
|
MPI |
Open MPI 4.1.5rc2 (Nvidia HPC-X v2.16) |
|
Compute VM SKU |
Standard_HB176-96rs_v4 (96 AMD EPYC™ 9V33X ("Genoa-X") CPU cores with AMD's 3D V-Cache, clock frequencies up to 3.7 GHz, 768 GB of RAM, 2.3 GB L3 cache, 400 Gb/s NDR InfiniBand |
|
Scheduler VM SKU |
Standard_D4ads_v5 |
CycleCloud Configuration
To create a Slurm Cluster, you must first ensure that CycleCloud is configured and running. Then, using the CycleCloud GUI, follow the instructions in the Create an HPC cluster from built-in templates guide to create a basic Slurm Cluster.
During the cluster creation process, most parameters can be left at their default values. However, there are a few that need to be updated based on your specific requirements. Under the "Required Settings" section of the Slurm Cluster creation screen, select the appropriate VM SKU for computation. For this demo, we are using Standard_D4ads_v5 for the scheduler and Standard_HB176-96rs_v4 as the HPC VM Type for compute. Additionally, set the Max HPC Cores to 192, as we will be running a 2 Node (2 x 96) MPI job with Infiniband in this test. If you are using more VMs, update the Max HPC cores accordingly.
In the “Advanced Settings” under “Slurm Settings” select 23.02.6-1 and in the “Software”, section select Ubuntu 22.04 LTS for all Machines.
We'll utilize CycleCloud's Cloud-Init feature to automate the installation of required dependencies for setting up OpenFOAM on Ubuntu 22.04. This script streamlines the installation process, guaranteeing that all necessary dependencies are installed accurately.
#cloud-config
runcmd:
- wget https://raw.githubusercontent.com/vinil-v/of2312-build-repo/main/install_deps_of2312.sh
- bash install_deps_of2312.sh
Save all the configurations and start the cluster.
Download and Compile OpenFOAM v2312 on Ubuntu 22.04
Engineers frequently prefer compiled versions of OpenFOAM over pre-compiled ones due to various benefits. These include the ability to optimize performance, access cutting-edge features, customize functionality, manage dependencies, optimize for HPC environments, and support educational and research endeavours. By compiling OpenFOAM from source, engineers can personalize the software to suit their precise requirements, leading to superior simulation outcomes. This customization ultimately drives advancements in computational fluid dynamics, facilitating groundbreaking discoveries in fluid dynamics and engineering applications.
To commence building OpenFOAM-v2312, access the Scheduler VM and clone the of2312-build-repo with the command:
git clone https://github.com/vinil-v/of2312-build-repo.git
This step is crucial for accessing the necessary files and initiating the building process.
You will find 4 scripts under of2312-build-repo
vinil@ofcls-prod-scheduler:~$ cd of2312-build-repo/
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ ls -l
total 20
-rw-rw-r-- 1 vinil vinil 65 Feb 12 15:47 README.md
-rw-rw-r-- 1 vinil vinil 2427 Feb 12 15:47 build_of2312-job.sh
-rw-rw-r-- 1 vinil vinil 1444 Feb 12 15:47 download_of2312.sh
-rw-rw-r-- 1 vinil vinil 1746 Feb 12 15:47 install_deps_of2312.sh
-rw-rw-r-- 1 vinil vinil 2755 Feb 12 15:47 motorbike_sim_job_of2312.sh
- download_of2312.sh: This script is responsible for downloading OpenFOAM-v2312 and extracting its contents. It automates the process of acquiring the necessary files, ensuring that they are readily available for the subsequent steps in the installation or compilation process.
- build_of2312-job.sh: This script facilitates the compilation of OpenFOAM-v2312 through a Slurm job. It sets up the necessary environment and configurations required for the compilation process, ensuring that it can be executed efficiently within a distributed computing environment managed by Slurm.
- motorbike_sim_job_of2312.sh: This script serves as a test job for running an OpenFOAM simulation using OpenFOAM-v2312. It provides a sample scenario or simulation setup, such as simulating a motorbike, to demonstrate the functionality and capabilities of the OpenFOAM software. This allows users to verify that the installation and compilation of OpenFOAM-v2312 was successful and that the software is functioning as expected for running simulations.
- install_deps_of2312.sh: Installs dependencies required for OpenFOAM-v2312 installation. This is used via cloud-init in cyclecloud.
Run the download_of2312.sh file to download and extract the OpenFOAM-v2312 version.
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ bash download_of2312.sh
Downloading OpenFOAM v2312...
Downloading ThirdParty package...
Extracting OpenFOAM v2312...
Extracting ThirdParty package...
Download and extraction completed.
The OpenFOAM compilation process on a 4-core node takes more than 5 hours, which is considerably long. Therefore, I've chosen not to run the compilation on a Scheduler node due to its limited resources and long compilation time. To speed up the compilation process, I've opted to run it as a Slurm job instead. Slurm will allocate a Standard_HB176-96rs_v4 node with 96 cores for this purpose. With this setup, the compilation job is completed in less than 20 minutes, significantly reducing the overall turnaround time and improving productivity. You could check the OpenFOAMv2312-Build-Job_<ID>.out file to check the progress of the compilation process.
Submit the build_of2312-job.sh using sbatch command to start the compilation process.
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ sbatch build_of2312-job.sh
Submitted batch job 2
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
2 hpc OpenFOAM vinil CF 0:04 1 ofcls-prod-hpc-1
Here is the Compilation slurm job script for the reference:
#!/bin/bash
# SLURM Directives
#SBATCH --job-name=OpenFOAMv2312-Build-Job
#SBATCH --ntasks=96
#SBATCH --output=%x_%j.out
#SBATCH --partition=hpc
# Load MPI and Prepare Software
module purge
module load mpi/hpcx
# Software Installation and Configuration
export INSTALL_DIR="$HOME/OpenFOAM"
# Compiling OpenFOAM
# Load OpenFOAM environment
. "$INSTALL_DIR/OpenFOAM-v2312/etc/bashrc"
cd "$INSTALL_DIR/OpenFOAM-v2312/"
# Set the number of compilation processes
export WM_NCOMPPROCS=$SLURM_NTASKS
# Start compilation
echo "Starting the compilation job at $(date)"
./Allwmake -j -s -q -l
echo "Finishing the compilation job at $(date)"
# Post-compilation Check
foamInstallationTest
Check the OpenFOAMv2312-Build-Job_<ID>.out log file for the Compilation job for any error. At the end of the job, it checks for foamInstallationTest to verify the compilation job.
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ tail -60 OpenFOAMv2312-Build-Job_2.out
Basic setup :
-------------------------------------------------------------------------------
OpenFOAM: OpenFOAM-v2312
ThirdParty: ThirdParty-v2312
Shell: bash
Host: ofcls-prod-hpc-1
OS: Linux version 5.15.0-1041-azure
-------------------------------------------------------------------------------
Main OpenFOAM env variables :
-------------------------------------------------------------------------------
Environment FileOrDirectory Valid Crit
-------------------------------------------------------------------------------
$WM_PROJECT_USER_DIR /shared/home/vinil/OpenFOAM/vinil-v2312 no no
$WM_THIRD_PARTY_DIR .../home/vinil/OpenFOAM/ThirdParty-v2312 yes maybe
$WM_PROJECT_SITE [env variable unset] no
-------------------------------------------------------------------------------
OpenFOAM env variables in PATH :
-------------------------------------------------------------------------------
Environment FileOrDirectory Valid Path Crit
-------------------------------------------------------------------------------
$WM_PROJECT_DIR ...ed/home/vinil/OpenFOAM/OpenFOAM-v2312 yes yes yes
$FOAM_APPBIN ...12/platforms/linux64GccDPInt32Opt/bin yes yes yes
$FOAM_SITE_APPBIN ...12/platforms/linux64GccDPInt32Opt/bin no no
$FOAM_USER_APPBIN ...12/platforms/linux64GccDPInt32Opt/bin no no
$WM_DIR ...e/vinil/OpenFOAM/OpenFOAM-v2312/wmake yes yes often
-------------------------------------------------------------------------------
OpenFOAM env variables in LD_LIBRARY_PATH :
-------------------------------------------------------------------------------
Environment FileOrDirectory Valid Path Crit
-------------------------------------------------------------------------------
$FOAM_LIBBIN ...12/platforms/linux64GccDPInt32Opt/lib yes yes yes
$FOAM_SITE_LIBBIN ...12/platforms/linux64GccDPInt32Opt/lib no no
$FOAM_USER_LIBBIN ...12/platforms/linux64GccDPInt32Opt/lib no no
$FOAM_EXT_LIBBIN ...v2312/platforms/linux64GccDPInt32/lib yes yes maybe
$MPI_ARCH_PATH ...-cuda12-gdrcopy2-nccl2.18-x86_64/ompi yes yes yes
-------------------------------------------------------------------------------
Software Components
-------------------------------------------------------------------------------
Software Version Location
-------------------------------------------------------------------------------
flex 2.6.4 /usr/bin/flex
make 4.3 /usr/bin/make
wmake 2312 /shared/home/vinil/OpenFOAM/OpenFOAM-v2312/wmake/wmake
gcc 11.4.0 /usr/bin/gcc
g++ 11.4.0 /usr/bin/g++
-------------------------------------------------------------------------------
icoFoam exists ...OAM-v2312/platforms/linux64GccDPInt32Opt/bin/icoFoam
Summary
-------------------------------------------------------------------------------
Base configuration ok.
Critical systems ok.
Done
Submitting OpenFOAM Motorbike example job
This job serves as a test job for running an OpenFOAM simulation using OpenFOAM-v2312. It provides a sample scenario or simulation setup, such as simulating a motorbike, to demonstrate the functionality and capabilities of the OpenFOAM software. This allows users to verify that the installation and compilation of OpenFOAM-v2312 was successful and that the software is functioning as expected for running simulations.
Submit the motorbike_sim_job_of2312.sh using sbatch to run the job in 2 x Standard_HB176-96rs_v4 with 192 CPUs.
Job submission script:
#!/bin/bash
# SLURM Directives
#SBATCH --job-name=OpenFOAM-MotorBike-Sim
#SBATCH --ntasks=192
#SBATCH --output=%x_%j.out
#SBATCH --partition=hpc
# Load MPI and OpenFOAM Environment
module purge
module load mpi/hpcx
export INSTALL_DIR="$HOME/OpenFOAM"
. "$INSTALL_DIR/OpenFOAM-v2312/etc/bashrc"
# Nodefile Generation
scontrol show hostname "$SLURM_JOB_NODELIST" | sort -u > "$HOME/nodefile-$SLURM_JOB_ID"
# Case Setup
mkdir "$HOME/run"
cp -r "$FOAM_TUTORIALS/incompressible/simpleFoam/motorBike" "$HOME/run"
cd "$HOME/run/motorBike"
# Parameter Setup and Simulation Run for 192 CPUs
cp "system/decomposeParDict.6" "system/decomposeParDict.$SLURM_NTASKS"
sed -i "s/numberOfSubdomains.*/numberOfSubdomains $SLURM_NTASKS;/" "system/decomposeParDict.$SLURM_NTASKS"
sed -i "s/ n (3 2 1)/ n (48 4 1);/" "system/decomposeParDict.$SLURM_NTASKS"
sed -i "s/(20 8 8)/(100 40 40)/" "system/blockMeshDict"
sed -i "s#system/decomposeParDict.6#system/decomposeParDict.$SLURM_NTASKS#" "Allrun"
mpiopts="$mpiopts -np $SLURM_NTASKS --hostfile $HOME/nodefile-$SLURM_JOB_ID"
./Allrun
Submit the job and check the output file for details.
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ sbatch motorbike_sim_job_of2312.sh
Submitted batch job 3
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ cat OpenFOAM-MotorBike-Sim_3.out
Loading mpi/hpcx
Loading requirement:
/opt/hpcx-v2.16-gcc-mlnx_ofed-ubuntu22.04-cuda12-gdrcopy2-nccl2.18-x86_64/modulefiles/hpcx
Running surfaceFeatureExtract on /shared/home/vinil/run/motorBike
Running blockMesh on /shared/home/vinil/run/motorBike
Running decomposePar on /shared/home/vinil/run/motorBike
Running snappyHexMesh (192 processes) on /shared/home/vinil/run/motorBike
Running topoSet (192 processes) on /shared/home/vinil/run/motorBike
Restore 0/ from 0.orig/ [processor directories]
Running patchSummary (192 processes) on /shared/home/vinil/run/motorBike
Running potentialFoam (192 processes) on /shared/home/vinil/run/motorBike
Running checkMesh (192 processes) on /shared/home/vinil/run/motorBike
Running simpleFoam (192 processes) on /shared/home/vinil/run/motorBike
vinil@ofcls-prod-scheduler:~/of2312-build-repo$
You can locate more detailed log files for each process OpenFOAM run within the case directory.
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ ls -l $HOME/run/motorBike/log.*
-rw-rw-r-- 1 vinil vinil 2407 Feb 12 17:42 /shared/home/vinil/run/motorBike/log.blockMesh
-rw-rw-r-- 1 vinil vinil 10549 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.checkMesh
-rw-rw-r-- 1 vinil vinil 95795 Feb 12 17:42 /shared/home/vinil/run/motorBike/log.decomposePar
-rw-rw-r-- 1 vinil vinil 2578 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.patchSummary
-rw-rw-r-- 1 vinil vinil 3021 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.potentialFoam
-rw-rw-r-- 1 vinil vinil 1572 Feb 12 17:47 /shared/home/vinil/run/motorBike/log.reconstructPar
-rw-rw-r-- 1 vinil vinil 52984 Feb 12 17:46 /shared/home/vinil/run/motorBike/log.reconstructParMesh
-rw-rw-r-- 1 vinil vinil 777220 Feb 12 17:46 /shared/home/vinil/run/motorBike/log.simpleFoam
-rw-rw-r-- 1 vinil vinil 276955 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.snappyHexMesh
-rw-rw-r-- 1 vinil vinil 4156 Feb 12 17:42 /shared/home/vinil/run/motorBike/log.surfaceFeatureExtract
-rw-rw-r-- 1 vinil vinil 1801 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.topoSet
vinil@ofcls-prod-scheduler:~/of2312-build-repo$
In conclusion, this blog has explored the seamless integration of OpenFOAM into high-performance computing environments facilitated by Azure CycleCloud. By configuring a Slurm-based HPC setup with HBv4 virtual machines and Ubuntu 22.04, engineers can optimize performance effectively. The post also delved into the crucial aspect of reducing OpenFOAM's compilation time and provided a comprehensive walkthrough for the compilation and execution process. Leveraging the parallel computing capabilities of MPI, along with Azure CycleCloud's scalable infrastructure, engineers can achieve remarkable simulation results. This demonstration underscores the immense potential of integrating OpenFOAM into HPC environments, heralding new horizons in fluid dynamics research and engineering applications.
References:
OpenFOAM Documentation
Azure CycleCloud Documentation
Create an HPC cluster from built-in templates
Azure HBv4 Series Virtual machine
Nvidia HPCX 2.16.2 toolkit
Azure Ubuntu-based HPC and AI Image