CUDA-Q issues on Gefion

Required prerequisites

• Consult the security policy. If reporting a security vulnerability, do not report the bug using this form. Use the process described in the policy to report the issue.
• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker to verify that this hasn't already been reported. +1 or comment there if it has.
• If possible, make a PR with a failing test to give us a starting point to work on!

Describe the bug

Documenting issues with running on Gefion.

The containerised workflow is as follows:

## Step 1 - download the container
enroot import docker://docker.gefion.dcai.dk\#nvidia/nightly/cuda-quantum:cu12-latest

## Step 2 - Generate the libcudaq\_distributed\_interface\_mpi.so
1. Start the container interactively:

srun --mpi=pmix --container-image=/dcai/users/chazoh/ansys/nvidia+nightly+cuda-quantum+cu12-latest.sqsh --container-mounts=/dcai:/dcai --pty bash


2. Need to edit the code and generate the file within the container

vim  /opt/nvidia/cudaq/distributed_interfaces/activate_custom_mpi.sh


3. Change so that the line 51 looks like:

$CXX -shared -std=c++17 -fPIC --disable-mlir-links \


4. Save changes and exit
5. Run: `/opt/nvidia/cudaq/distributed_interfaces/activate_custom_mpi.sh` to generate the `libcudaq_distributed_interface_mpi.so` file
6. Move the file to your home `mv /opt/nvidia/cudaq/distributed_interfaces/libcudaq_distributed_interface_mpi.so ~/`
7. Exit the interactive job

After these steps, the slurm script is:

#!/bin/bash

#SBATCH --nodes 32 --nodelist=dgx[001-040]
#SBATCH --gpus-per-node 8
#SBATCH --mem 0
#SBATCH --ntasks-per-node 8
#SBATCH --time 24:00:00
#SBATCH --output=%j.txt
#SBATCH --array=1-10

export NCCL_SOCKET_IFNAME=ens6f0
export NCCL_IB_HCA=mlx5_0:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_9:1,mlx5_10:1,mlx5_11:1
export UCX_NET_DEVICES=mlx5_0:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_9:1,mlx5_10:1,mlx5_11:1
export SHARP_COLL_ENABLE_PCI_RELAXED_ORDERING=1
export OMPI_MCA_pml=ucx
export OMPI_MCA_btl=^vader,tcp,openib,uct
export OMPI_MCA_coll_hcoll_enable=0
export NCCL_COLLNET_ENABLE=0
export NCCL_SHARP_DISABLE=1

echo "NODELIST="${SLURM_NODELIST}
echo "Running SLURM_ARRAY_TASK_ID=$SLURM_ARRAY_TASK_ID"
echo "Start timestamp: $(date -Iseconds)"
srun --mpi=pmix --container-image=/dcai/users/chazoh/ansys/nvidia+nightly+cuda-quantum+cu12-latest.sqsh --container-mounts=/dcai/:/dcai/,/cm/:/cm/ bash -c "(export CUDAQ_MPI_COMM_LIB=/dcai/users/chazoh/ansys/libcudaq_distributed_interface_mpi.so; python /dcai/users/chazoh/ansys/mgpu.py)"
echo "Finish timestamp: $(date -Iseconds)"

where we execute a basic mgpu file:

import cudaq

cudaq.set_target("nvidia", option="mgpu,fp64")
cudaq.mpi.initialize()
num_ranks = cudaq.mpi.num_ranks()
rank = cudaq.mpi.rank()


qubit_count = 41
print('num_ranks', num_ranks, 'current rank', rank, 'qubits', qubit_count)

@cudaq.kernel
def kernel(qubit_count: int):
    qubits = cudaq.qvector(qubit_count)
    h(qubits[0])
    for i in range(1, qubit_count):
        cx(qubits[0], qubits[i])
        
counts = cudaq.sample(kernel, qubit_count)

if (cudaq.mpi.rank()==0):
    counts.dump()

The observed behaviour is that the probability of failure increases with number of nodes. Smaller jobs are successful however larger jobs of around 32 nodes seem to fail about 20% of the time. i.e. 2/10 jobs will produce the following error:

NODELIST=dgx[001-002,120-149]
Running SLURM_ARRAY_TASK_ID=6
Start timestamp: 2025-05-21T14:58:41+02:00
[dgx149:1219315:0:1219315] ib_mlx5_log.c:171  Remote access on mlx5_0:1/IB (synd 0x13 vend 0x88 hw_synd 0/0)
[dgx149:1219315:0:1219315] ib_mlx5_log.c:171  RC QP 0x425d wqe[23]: RDMA_WRITE s-- [rva 0x15513b002800 rkey 0x203371] [va 0x155136002800 len 16777216 lkey 0x203ac6] [rqpn 0x6272 dlid=1898 sl=0 port=1 src_path_bits=0]
==== backtrace (tid:1219315) ====
 0  /usr/local/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x1551ef46e574]
 1  /usr/local/ucx/lib/libucs.so.0(ucs_fatal_error_message+0xb6) [0x1551ef46c1b6]
 2  /usr/local/ucx/lib/libucs.so.0(ucs_log_default_handler+0x835) [0x1551ef46fb05]
 3  /usr/local/ucx/lib/libucs.so.0(ucs_log_dispatch+0xe4) [0x1551ef46fe64]
 4  /usr/local/ucx/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_completion_with_err+0x60f) [0x1551ee91ea2f]
 5  /usr/local/ucx/lib/ucx/libuct_ib.so.0(+0x39274) [0x1551ee932274]
 6  /usr/local/ucx/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_check_completion+0x35) [0x1551ee91fa55]
 7  /usr/local/ucx/lib/ucx/libuct_ib.so.0(+0x3ac22) [0x1551ee933c22]
 8  /usr/local/ucx/lib/libucp.so.0(ucp_worker_progress+0x5a) [0x1551ef06c10a]
 9  /usr/local/openmpi/lib/libopen-pal.so.40(opal_progress+0x34) [0x1551eed456e4]
10  /usr/local/openmpi/lib/libmpi.so(ompi_request_default_wait_all+0x10d) [0x1551ef133bad]
11  /usr/local/openmpi/lib/libmpi.so(PMPI_Waitall+0x97) [0x1551ef1714d7]
12  /opt/nvidia/cudaq/lib/libnvqir-nvidia-mgpu.so(+0xefc68) [0x155497000c68]
13  /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(+0x440bcb) [0x1554dde40bcb]
14  /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(+0x43ef4f) [0x1554dde3ef4f]
15  /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(+0x104371) [0x1554ddb04371]
16  /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(custatevecSVSwapWorkerExecute+0x578) [0x1554ddaffc98]
17  /opt/nvidia/cudaq/lib/libnvqir-nvidia-mgpu.so(+0x1582d1) [0x1554970692d1]
18  /opt/nvidia/cudaq/lib/libnvqir-nvidia-mgpu.so(+0x1587ce) [0x1554970697ce]
19  /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xdf04e) [0x1554adc3304e]
20  /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xdf14b) [0x1554adc3314b]
21  /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xdf50e) [0x1554adc3350e]
22  /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xddd4f) [0x1554adc31d4f]
23  /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(_ZN5cudaq23CusvsimCircuitSimulatorIdE19flushGateApplicatorEb+0x47d) [0x1554adbdb11d]
24  /opt/nvidia/cudaq/lib/libnvqir-custatevec-fp64.so(_ZN5nvqir20CircuitSimulatorBaseIdE21resetExecutionContextEv+0x458) [0x1554e2435a78]
25  /opt/nvidia/cudaq/lib/libcudaq-em-default.so(+0x33b4f) [0x1554e370bb4f]
26  /opt/nvidia/cudaq/lib/libcudaq-platform-default.so(+0xa6dcb) [0x1554e2aefdcb]
27  /opt/nvidia/cudaq/lib/libcudaq.so(_ZN5cudaq16quantum_platform14reset_exec_ctxEm+0x16) [0x1554e37d6546]
28  /opt/nvidia/cudaq/cudaq/mlir/_mlir_libs/_quakeDialects.cpython-310-x86_64-linux-gnu.so(+0x2ccc47) [0x1554e3bedc47]
29  /opt/nvidia/cudaq/cudaq/mlir/_mlir_libs/_quakeDialects.cpython-310-x86_64-linux-gnu.so(+0x2c2eaa) [0x1554e3be3eaa]
30  python(+0x18ae12) [0x5555556dee12]
31  python(_PyObject_MakeTpCall+0x25b) [0x5555556d575b]
32  python(_PyEval_EvalFrameDefault+0x5f66) [0x5555556cf1d6]
33  python(_PyFunction_Vectorcall+0x7c) [0x5555556df66c]
34  python(_PyEval_EvalFrameDefault+0x5642) [0x5555556ce8b2]
35  python(+0x259f56) [0x5555557adf56]
36  python(PyEval_EvalCode+0x86) [0x5555557ade26]
37  python(+0x280808) [0x5555557d4808]
38  python(+0x27b00f) [0x5555557cf00f]
39  python(+0x2805a5) [0x5555557d45a5]
40  python(_PyRun_SimpleFileObject+0x1a8) [0x5555557d3b88]
41  python(_PyRun_AnyFileObject+0x47) [0x5555557d3867]
42  python(Py_RunMain+0x2be) [0x5555557c7e5e]
43  python(Py_BytesMain+0x2d) [0x5555557a1e6d]
44  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x1555551d7d90]
45  /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x1555551d7e40]
46  python(_start+0x25) [0x5555557a1d65]
=================================
[dgx149:1219315] *** Process received signal ***
[dgx149:1219315] Signal: Aborted (6)
[dgx149:1219315] Signal code:  (-6)
[dgx149:1219315] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x1555551f0520]
[dgx149:1219315] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x1555552449fc]
[dgx149:1219315] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x1555551f0476]
[dgx149:1219315] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x1555551d67f3]
[dgx149:1219315] [ 4] /usr/local/ucx/lib/libucs.so.0(+0x271bb)[0x1551ef46c1bb]
[dgx149:1219315] [ 5] /usr/local/ucx/lib/libucs.so.0(ucs_log_default_handler+0x835)[0x1551ef46fb05]
[dgx149:1219315] [ 6] /usr/local/ucx/lib/libucs.so.0(ucs_log_dispatch+0xe4)[0x1551ef46fe64]
[dgx149:1219315] [ 7] /usr/local/ucx/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_completion_with_err+0x60f)[0x1551ee91ea2f]
[dgx149:1219315] [ 8] /usr/local/ucx/lib/ucx/libuct_ib.so.0(+0x39274)[0x1551ee932274]
[dgx149:1219315] [ 9] /usr/local/ucx/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_check_completion+0x35)[0x1551ee91fa55]
[dgx149:1219315] [10] /usr/local/ucx/lib/ucx/libuct_ib.so.0(+0x3ac22)[0x1551ee933c22]
[dgx149:1219315] [11] /usr/local/ucx/lib/libucp.so.0(ucp_worker_progress+0x5a)[0x1551ef06c10a]
[dgx149:1219315] [12] /usr/local/openmpi/lib/libopen-pal.so.40(opal_progress+0x34)[0x1551eed456e4]
[dgx149:1219315] [13] /usr/local/openmpi/lib/libmpi.so(ompi_request_default_wait_all+0x10d)[0x1551ef133bad]
[dgx149:1219315] [14] /usr/local/openmpi/lib/libmpi.so(PMPI_Waitall+0x97)[0x1551ef1714d7]
[dgx149:1219315] [15] /opt/nvidia/cudaq/lib/libnvqir-nvidia-mgpu.so(+0xefc68)[0x155497000c68]
[dgx149:1219315] [16] /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(+0x440bcb)[0x1554dde40bcb]
[dgx149:1219315] [17] /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(+0x43ef4f)[0x1554dde3ef4f]
[dgx149:1219315] [18] /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(+0x104371)[0x1554ddb04371]
[dgx149:1219315] [19] /usr/local/lib/python3.10/dist-packages/cuquantum/lib/libcustatevec.so.1(custatevecSVSwapWorkerExecute+0x578)[0x1554ddaffc98]
[dgx149:1219315] [20] /opt/nvidia/cudaq/lib/libnvqir-nvidia-mgpu.so(+0x1582d1)[0x1554970692d1]
[dgx149:1219315] [21] /opt/nvidia/cudaq/lib/libnvqir-nvidia-mgpu.so(+0x1587ce)[0x1554970697ce]
[dgx149:1219315] [22] /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xdf04e)[0x1554adc3304e]
[dgx149:1219315] [23] /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xdf14b)[0x1554adc3314b]
[dgx149:1219315] [24] /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xdf50e)[0x1554adc3350e]
[dgx149:1219315] [25] /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(+0xddd4f)[0x1554adc31d4f]
[dgx149:1219315] [26] /opt/nvidia/cudaq/lib/libnvqir-cusvsim-fp64.so(_ZN5cudaq23CusvsimCircuitSimulatorIdE19flushGateApplicatorEb+0x47d)[0x1554adbdb11d]
[dgx149:1219315] [27] /opt/nvidia/cudaq/lib/libnvqir-custatevec-fp64.so(_ZN5nvqir20CircuitSimulatorBaseIdE21resetExecutionContextEv+0x458)[0x1554e2435a78]
[dgx149:1219315] [28] /opt/nvidia/cudaq/lib/libcudaq-em-default.so(+0x33b4f)[0x1554e370bb4f]
[dgx149:1219315] [29] /opt/nvidia/cudaq/lib/libcudaq-platform-default.so(+0xa6dcb)[0x1554e2aefdcb]
[dgx149:1219315] *** End of error message ***
/usr/bin/bash: line 1: 1219315 Aborted                 (core dumped) python /dcai/users/chazoh/ansys/mgpu.py
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx149 [31]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:254]
srun: Job step aborted: Waiting up to 1802 seconds for job step to finish.
slurmstepd: error: *** STEP 50357.0 ON dgx001 CANCELLED AT 2025-05-21T14:59:36 ***
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx134 [16]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:130]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx142 [24]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:199]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx128 [10]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:83]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx144 [26]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:210]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx126 [8]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:70]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx143 [25]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:204]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx139 [21]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:169]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx149 [31]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:253]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx001 [0]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:6]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx132 [14]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:118]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx138 [20]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:166]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx146 [28]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:228]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx145 [27]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:219]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx121 [3]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:31]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx130 [12]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:102]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx133 [15]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:123]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx131 [13]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:105]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx135 [17]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:136]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx147 [29]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:235]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx140 [22]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:179]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx136 [18]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:147]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx137 [19]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:156]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx120 [2]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:17]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx129 [11]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:94]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx124 [6]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:50]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx122 [4]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:38]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx148 [30]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:244]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx123 [5]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:44]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx141 [23]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:185]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx127 [9]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:77]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx125 [7]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:58]
slurmstepd: error:  mpi/pmix_v4: _errhandler: dgx002 [1]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.50357.0:11]
srun: error: dgx149: tasks 248-253,255: Killed
srun: error: dgx149: task 254: Exited with exit code 134
srun: error: dgx123: tasks 40-47: Killed
srun: error: dgx140: tasks 176-183: Killed
srun: error: dgx137: tasks 152-159: Killed
srun: error: dgx120: tasks 16-23: Killed
srun: error: dgx136: tasks 144-151: Killed
srun: error: dgx148: tasks 240-247: Killed
srun: error: dgx144: tasks 208-215: Killed
srun: error: dgx147: tasks 232-239: Killed
srun: error: dgx135: tasks 136-143: Killed
srun: error: dgx130: tasks 96-103: Killed
srun: error: dgx125: tasks 56-63: Killed
srun: error: dgx145: tasks 216-223: Killed
srun: error: dgx134: tasks 128-135: Killed
srun: error: dgx129: tasks 88-95: Killed
srun: error: dgx121: tasks 24-31: Killed
srun: error: dgx146: tasks 224-231: Killed
srun: error: dgx138: tasks 160-167: Killed
srun: error: dgx142: tasks 192-199: Killed
srun: error: dgx128: tasks 80-87: Killed
srun: error: dgx133: tasks 120-127: Killed
srun: error: dgx141: tasks 184-191: Killed
srun: error: dgx132: tasks 112-119: Killed
srun: error: dgx126: tasks 64-71: Killed
srun: error: dgx001: tasks 0-7: Killed
srun: error: dgx131: tasks 104-111: Killed
srun: error: dgx127: tasks 72-79: Killed
srun: error: dgx143: tasks 200-207: Killed
srun: error: dgx122: tasks 32-39: Killed
srun: error: dgx124: tasks 48-55: Killed
srun: error: dgx002: tasks 8-15: Killed
srun: error: dgx139: tasks 168-175: Killed
Finish timestamp: 2025-05-21T14:59:41+02:00

If however, we use the pip install recipe on the machine:

module load GCCcore/13.3.0 HPCX

python -m pip install --upgrade --no-cache-dir --use-deprecated=legacy-resolver cudaq

export MPI_PATH=$HPCX_MPI_DIR

export CXX=mpicxx

bash `find ~/ -name 'activate_custom_mpi.sh'`

and execute via the slurm script, it works without any failures.

cc @pioch-02 @mitchdz

Steps to reproduce the bug

NA

Expected behavior

NA

Is this a regression? If it is, put the last known working version (or commit) here.

Not a regression

Environment

• CUDA-Q version:
• Python version:
• C++ compiler:
• Operating system:

Suggestions

No response

Start with distributed_interfaces/activate_custom_mpi.sh and the provided mgpu.py workload, then run the supplied SLURM container workflow at smaller and approximately 32-node scales. Compare the intermittent UCX/OpenMPI remote-access failure and determine the affected CUDA-Q distributed MPI entry point; done means the workload runs reliably at the reported scale or the failure is reproduced with a validated fix.