Test CUDA failure common.cu:941 'invalid device ordinal' when test two nodes with nvhpc

I compile nccl-tests with the command:

make MPI=1 MPI_HOME=${NVHPC_ROOT}/comm_libs/12.4/hpcx/hpcx-2.19/ompi NCCL_HOME=${NVHPC_ROOT}/comm_libs/nccl CUDA_HOME=${NVHPC_ROOT}/cuda

And run the command to test the all_reduce_perf:

mpirun --host g0010:8,g0016:8 \
-x LD_LIBRARY_PATH=${NVHPC_ROOT}/comm_libs/nccl/lib:${NVHPC_ROOT}/cuda/lib64 \
-x PATH=${NVHPC_ROOT}/comm_libs/mpi/bin:${NVHPC_ROOT}/compilers/bin:$PATH \
-x NCCL_SOCKET_IFNAME=ib0 \
-x NCCL_DEBUG=INFO \
-x NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH \
-x NCCL_PXN_DISABLE=0 \
-x NCCL_IB_QPS_PER_CONNECTION=4 \
-x NCCL_ALGO=Ring \
-x NCCL_IB_DISABLE=0 \
-x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_6,mlx5_7,mlx5_8,mlx5_9 \
/home/clouduser/nccl-tests/build/all_reduce_perf -b 128M -e 2G -f 2 -g 8

And I got the error:

# Using devices
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888386: Test failure common.cu:891
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888385: Test failure common.cu:891
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888382: Test failure common.cu:891
g0016: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0016 pid 3645592: Test failure common.cu:891
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888383: Test failure common.cu:891
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888387: Test failure common.cu:891
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888384: Test failure common.cu:891
g0010: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0010 pid 888388: Test failure common.cu:891
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
g0016: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0016 pid 3645590: Test failure common.cu:891
g0016: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0016 pid 3645594: Test failure common.cu:891
g0016: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0016 pid 3645588: Test failure common.cu:891
g0016: Test CUDA failure common.cu:941 'invalid device ordinal'
 .. g0016 pid 3645593: Test failure common.cu:891
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[21831,1],5]
  Exit code:    2

NOTE: When I directly use openmpi instead of nvhpc, the test run successfully.

Start by reproducing the reported two-node mpirun command with the NVIDIA HPC SDK and compare it with the working Open MPI setup. Inspect common.cu around lines 941 and 891, then determine why the nvhpc run selects an invalid device ordinal; done means all_reduce_perf completes across both nodes without the CUDA failure.