Multi-GPU not working with backend tensornet and target is remote-mqpu

Required prerequisites

• Consult the security policy. If reporting a security vulnerability, do not report the bug using this form. Use the process described in the policy to report the issue.
• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker to verify that this hasn't already been reported. +1 or comment there if it has.
• If possible, make a PR with a failing test to give us a starting point to work on!

Describe the bug

Details of the error while using multi-GPU:

I tried executing the following example given in this documentation
multi-gpu example

As per the Documentation I set the target to remote-mqpu , backend=tensornet

Issue:- The processes are running on a single GPU and not Multiple GPUS

As the above thing did not work I also tried doing the following settings

I also tried this as per the documentation for the tensornet backend and target to remote-qpu , following all the steps mentioned

Issue:- Every time while executing the following line I am getting a error saying core dumped  error

CUDA_VISIBLE_DEVICES=0,1 mpiexec -np 2 python3 "$qpud_py" --port 5001

Steps to reproduce the bug

import cudaq
import traceback
backend="tensornet"
urls = "localhost:5001,localhost:5002"
servers = "2"

Define a kernel to be sampled.

@cudaq.kernel
def kernel(controls_count: int):
controls = cudaq.qvector(controls_count)
targets = cudaq.qvector(2)
# Place controls in superposition state.
h(controls)
for target in range(2):
x.ctrl(controls, targets[target])
# Measure.
mz(controls)
mz(targets)

Set the target to execute on and query the number of QPUs in the system;

The number of QPUs is equal to the number of (auto-)launched server instances.

cudaq.set_target("remote-mqpu",
backend=backend,
auto_launch="",
url=urls)
qpu_count = cudaq.get_target().num_qpus()
print("Number of virtual QPUs:", qpu_count)

We will launch asynchronous sampling tasks,

and will store the results as a future we can query at some later point.

Each QPU (indexed by an unique Id) is associated with a remote REST server.

count_futures = []

for _ in range(100):

for i in range(qpu_count):
try:
result = cudaq.sample_async(kernel, i+1, qpu_id=i)
except Exception as e:
print(f"Error launching sample_async for QPU {i}: {e}")
continue
count_futures.append(result)
print("Sampling jobs launched for asynchronous processing.")

Go do other work, asynchronous execution of sample tasks on-going.

Get the results, note future::get() will kick off a wait

if the results are not yet available.

print("Waiting for results...")
for idx in range(len(count_futures)):
try:
counts = count_futures[idx].get()
print(counts)
except Exception as e:
print(e)
print(f"Error getting result for QPU {idx}: {e}")
traceback.print_exc()
'''

Expected behavior

It should work on multiple gpus but code is giving error Aborted (core dumped)

Is this a regression? If it is, put the last known working version (or commit) here.

Not a regression

Environment

• CUDA-Q version:
• Python version:
• C++ compiler:
• Operating system:

Suggestions

No response

Start with the linked multi-GPU and tensornet documentation, then run the provided Python script with the remote-mqpu target and the CUDA_VISIBLE_DEVICES=0,1 mpiexec command. Check whether the two configured server URLs launch on separate GPUs and capture the full core-dump error; done means the sample completes using multiple GPUs.