CUDA-aware MPI

CUDA aware MPI allows OpenMPI to handle CUDA communication between GPUs that are not necessarily peers. For more information about GPU peers, please refer to Understanding GPU peer access.

Operating system support

CUDA-aware MPI is only supported on Linux.

Microsoft Windows does not support CUDA-aware MPI.

When is CUDA-aware MPI required?

This feature is required when you want to run simulations on GPUs that are not peers. If the GPUs are not connected via NVLINK, you will likely want to compile OpenMPI with CUDA support.

If you see this error from the solver, Error: trying to use MPI for GPU-GPU communication but MPI build is not CUDA-aware, this means you will need OpenMPI with CUDA support.

Check if MPI is CUDA aware

Run the command:

ompi_info --parsable --all | grep mpi_built_with_cuda_support:value

If your installation support is CUDA-aware, you will see the following:

mca:mpi:base:param:mpi_built_with_cuda_support:value:true

If your installation support is not CUDA-aware, you will see the following:

mca:mpi:base:param:mpi_built_with_cuda_support:value:false

How do I add CUDA support to OpenMPI?

You need to recompile OpenMPI with CUDA support. When you configure OpenMPI, add the --with-cuda argument. If you have multiple CUDA toolkit installations, be sure to compile with the same version as your currently running NVidia driver. For example: ./configure --with-cuda=/usr/local/cuda-12.1 will compile OpenMPI with this specific CUDA installation.

For more information on OpenMPI compilation, please refer to Installation.