Windows

Setup

These instructions are intended for M-Star CFD installation on a Windows desktop or Windows server with a multi-GPU environment.

Software Prerequisites

  • NVidia display driver version 418.96 or newer. We recommend having the latest version of your display driver installed.

  • Microsoft MPI: Latest version here.

Hardware Prerequisites

  • X86 64-bit computer

  • NVlink (A wire-based communications link which allows several GPUs to interact directly through a high-speed interconnection)

  • Supported GPU, including: NVIDIA® Tesla™ / Quadro K-Series or Tesla™ / Quadro™ P-Series GPU

Accessing the Software

Go to the M-Star CFD Download Portal to access the software.

Installation

Run the MSI file downloaded from the portal.

Multi-GPU Setup

To execute multi-GPU jobs on Windows, the GPUs must support a feature called TCC Mode. Generally this feature is limited to NVidia Quadro, Tesla, and TITAN products. Geforce tends not to have this feature. You should independently verify this specification if you intend to run multi-GPU jobs on Windows.

Additionally, TCC mode requires dedicated usage for CUDA jobs, and may not be used for display. For example, you cannot put 2x V100 GPUs in TCC mode and plug one of them into your monitor. You must have a third GPU dedicated for display in that case. The third GPU should be the same type as the other GPUs in the system.

The following must be in place for Windows to run a multi-GPU setup:

  • The GPU itself supports TCC mode (generally limited to Quadro and Tesla products).

  • The GPUs used in simulation must be in TCC mode.

  • The monitor is not connected to any TCC Mode GPU.

  • All GPUs in the system should be the same type (do not mix different GPUs for calculation and display because this can cause system instability and driver issues).

  • BIOS setting for primary display adapter is set to a GPU not in TCC mode.

  • Microsoft MPI is installed.

When using multiple GPUs for M-Star on Windows, you will need an auxiliary GPU dedicated to running the desktop environment. TCC Mode disables the graphics capability of the GPU, putting the GPU into a mode where only CUDA calculations may take place. This means that you may not connect the monitor to any GPU that is in TCC mode.

good gpu setup

This is an example of a good multi-GPU setup on Windows.

In the figure above, GPUs 1 and 2 would be set up in TCC Mode and used for M-Star simulations. GPU 3 would be dedicated to desktop display purposes and left in the default WDDM Mode.

Important

Multi-GPU jobs on Windows require TCC Mode. TCC Mode GPUs must only be used for CUDA jobs and may not be used for display purposes. BIOS setting for primary display adapter must be set on a non-TCC mode GPU.

Set primary display adapter in BIOS

Some hardware configurations require this setting. If you run into operating system stability issues or unexplained blue screen failures in Windows, this may be something to check.

  1. Restart your computer and open the BIOS configuration.

  2. Navigate to the primary display adapter setting.

  3. Change the setting to a GPU that is not set to TCC mode (In the example above, this would be set to the GPU 3).

  4. Save settings and reboot.

Set GPUs to TCC Mode

To set the GPUs in TCC Mode:

  1. Open a command window in Administrator mode.

  2. Identify where the nvidia-smi utility is installed. It is typically located at one of the following locations:

    • "C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi"

    • "C:\Windows\System32\nvidia-smi"

  3. In the command window, enter the following (note the quotes around the path). This shows all the GPUs in your machine. Take note of the GPUs you want to set in TCC mode.

    "C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi" -L

  4. For each GPU you want to place in TCC mode, enter the following. Replace ID_HERE with the index from the previous step.

    "C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi" -g ID_HERE -dm 1

The -dm 1 argument sets the GPU in TCC mode. The -g 0 argument selects GPU index 0 from step (3). Use the argument -dm 0 to place a GPU back into the default WDDM mode.

  1. Reboot the computer.

  2. Open the command window again (cmd.exe) and run the following to verify. Ensure the configured GPUs show as being in TCC mode.

    "C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi" --query-gpu=index,name,driver_model.current --format=csv

More information on TCC mode is available here.

Multi-Node Setup

Multi-node jobs that span multiple computers are currently not supported on Windows. This type of workload is only supported under Linux.

Troubleshooting Pre and Post startup

See Troubleshooting M-Star Pre/Post start up for additional information