QwenVL Training Error When Using Specific GPUs on a Single Server

I want to train the QwenVL model using GPU 2 and GPU 3 on a single server. I modified the YAML configuration to use two GPUs and set the device mapping accordingly. At the same time, I specified CUDA_VISIBLE_DEVICES as 2 and 3 in the shell script.

However, I encountered the following problems:

1. The program still runs on GPU 0 and GPU 1, not on GPU 2 and GPU 3 as intended.

2. The inference processes (actor_infer-0 and actor_infer-1) both report errors related to CUDA, saying there is an "invalid argument" and failing during collective RPC.

3. Meanwhile, the training processes (actor_train-0 and actor_train-1) do not report any error and run normally.

Could you please help me understand why:

The inference actors are failing;

The training still uses GPU 0 and 1 even though I tried to assign GPU 2 and 3?

Thanks a lot!

[yaml.txt](https://github.com/user-attachments/files/21013280/yaml.txt)

Start with the attached yaml.txt and the shell script that sets CUDA_VISIBLE_DEVICES=2,3, then inspect the actor_infer and actor_train logs. Reproduce the single-server two-GPU configuration and compare logical device numbering with the configured device mapping. Done means explaining the inference CUDA invalid-argument failure and why training selects GPUs 0 and 1, with a verified configuration or documented limitation.

Written by the indexing model from the issue text.