`mgpu` does not work as expected with initialize statevector functionality

Required prerequisites

• Consult the security policy. If reporting a security vulnerability, do not report the bug using this form. Use the process described in the policy to report the issue.
• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker to verify that this hasn't already been reported. +1 or comment there if it has.
• If possible, make a PR with a failing test to give us a starting point to work on!

Describe the bug

The workflow below works with 1 and 8 gpus:

import cudaq

cudaq.set_target('nvidia', option='mgpu,fp64')

num_qubits = 30
@cudaq.kernel
def kernel():
    qubits = cudaq.qvector(num_qubits)

result = cudaq.sample(kernel)
print('done')

mpirun -np 1 --allow-run-as-root python3 test.py --cudaq-full-stack-trace

mpirun -np 8 --allow-run-as-root python3 test.py --cudaq-full-stack-trace

allocating more gpus than the problem needs does not throw an error. This is not the case for the workflow below:

# !pip install cupy-cuda12x

import cudaq
import cupy as cp 

cudaq.set_target('nvidia', option='mgpu,fp64')

num_qubits = 30

initialize_statvec = cp.zeros(2**num_qubits, dtype=cp.complex128)
initialize_statvec[0] = 1./cp.sqrt(2.0)

initial_state = cudaq.State.from_data(initialize_statvec)

@cudaq.kernel
def kernel(state: cudaq.State):
    qubits = cudaq.qvector(state)

result = cudaq.sample(kernel, initial_state)
print('done')

1 GPU works fine:

mpirun -np 1 --allow-run-as-root python3 test.py --cudaq-full-stack-trace

This throws an error:

mpirun -np 8 --allow-run-as-root python3 test.py --cudaq-full-stack-trace

Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
Traceback (most recent call last):
  File "/home/cudaq_work/ansys/test.py", line 21, in <module>
    result = cudaq.sample(kernel, initial_state)
  File "/opt/nvidia/cudaq/cudaq/runtime/sample.py", line 86, in sample
    kernel(*args)
  File "/opt/nvidia/cudaq/cudaq/kernel/kernel_decorator.py", line 497, in __call__
    cudaq_runtime.pyAltLaunchKernel(self.name,
RuntimeError: [addQubitsToState] Incompatible number of local wires 30, expected 27
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996962] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996964] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996963] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996968] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996965] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996961] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996966] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** The MPI_Barrier() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[fi-kermit:3996967] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[35385,1],7]
  Exit code:    1
--------------------------------------------------------------------------

Steps to reproduce the bug

NA

Expected behavior

NA

Is this a regression? If it is, put the last known working version (or commit) here.

Not a regression

Environment

• CUDA-Q version:
• Python version:
• C++ compiler:
• Operating system:

Suggestions

No response

Reproduce the two workflows using cudaq.set_target('nvidia', option='mgpu,fp64'), cudaq.State.from_data, and mpirun with 1 and 8 processes. Trace the addQubitsToState path and the reported local-wire mismatch, then verify that the initialized-state workflow completes across 8 GPUs without the MPI_FINALIZE errors.