CUDA_VISIBLE_DEVICES isn't correctly inherited on a SLURM system

Describe the bug
This issue occurs on a SLURM cluster where worker nodes equipped with multiple GPU's are shared amongst users. GPU's are given slot number assignments (for example, on a node with 8 GPU's:0-7), and users may be assigned any number of the GPU's of a node by the SLURM scheduler. For example, a SLURM assignment could set CUDA_VISIBLE_DEVICES to 4,5.

Per this bug report, I tried to use the --include flag with the deepspeed command to input my specific GPU numeric assignments (i.e., the values of CUDA_VISIBLE_DEVICES) from SLURM. When I tried the following command:

deepspeed --include localhost:4,5 mycode.py --deepspeed ds_config.json

I receive the following errors:

[2021-08-27 14:16:05,380] [WARNING] [runner.py:122:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Traceback (most recent call last):
  File "/envs/huggingface_deepspeed_python/bin/deepspeed", line 6, in <module>
    main()
  File "/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed/launcher/runner.py", line 280, in main
    active_resources = parse_inclusion_exclusion(resource_pool,
  File "/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed/launcher/runner.py", line 248, in parse_inclusion_exclusion
    return parse_resource_filter(active_resources,
  File "envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed/launcher/runner.py", line 198, in parse_resource_filter
    raise ValueError("No slot '{}' specified on host '{}'".format(
ValueError: No slot '4' specified on host 'localhost'

Note: the value of echo $CUDA_VISIBLE_DEVICES was 4,5 in this example.

When I instead tried the following code:

deepspeed --include localhost:0,1 mycode.py --deepspeed ds_config.json

I successfully ran the mycode.py script with the following output preceding it:

[2021-08-27 14:16:11,581] [WARNING] [runner.py:122:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-08-27 14:16:11,717] [INFO] [runner.py:360:main] cmd = /envs/huggingface_deepspeed_python/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 mycode.py --deepspeed ds_config.json
[2021-08-27 14:16:12,341] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2021-08-27 14:16:12,341] [INFO] [launch.py:86:main] nnodes=1, num_local_procs=2, node_rank=0
[2021-08-27 14:16:12,341] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2021-08-27 14:16:12,341] [INFO] [launch.py:102:main] dist_world_size=2
[2021-08-27 14:16:12,341] [INFO] [launch.py:104:main] Setting CUDA_VISIBLE_DEVICES=0,1

But as I received GPU's that were already in use, my processes ran out of memory. I.e.,

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 31.75 GiB total capacity; 550.30 MiB already allocated; 2.75 MiB free; 574.00 MiB reserved in total by PyTorch)

After reading the bug report I referenced above, I noticed the world_info dictionary that's submitted as a base-64 string to the launch script uses torch.cuda.device_count() to create the list of GPU's that should be used. Would it instead be possible to inherit the pre-existing CUDA_VISIBLE_DEVICES assignments?

As a preliminary work-around, I've been creating a custom world_info dictionary and calling the launch script directly, like so:

WID=`echo  {\"localhost\": [$CUDA_VISIBLE_DEVICES]} | base64`
python -u -m deepspeed.launcher.launch --world_info=$WID --master_addr=127.0.0.1 --master_port=29500 mycode.py --deepspeed ds_config.json

Hostfile attempt

I also tried with a specified hostfile. This input:

$ cat myhostfile
workerhostname slots=4,5

resulted in the following output:

[2021-08-27 15:13:12,639] [ERROR] [runner.py:139:fetch_hostfile] Hostfile is not formatted correctly, unable to proceed with training.
Traceback (most recent call last):
  File "/envs/huggingface_deepspeed_python/bin/deepspeed", line 6, in <module>
    main()
  File "/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed/launcher/runner.py", line 267, in main
    resource_pool = fetch_hostfile(args.hostfile)
  File "/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed/launcher/runner.py", line 141, in fetch_hostfile
    raise err
  File "/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed/launcher/runner.py", line 137, in fetch_hostfile
    slot_count = int(slot_count)
ValueError: invalid literal for int() with base 10: '4,5'

Called via:

deepspeed --hostfile myhostfile mycode.py --deepspeed ds_config.json

I presumed I was simply misunderstanding the syntax, so as an experiment I changed the hostfile to:

workerhostname slots=5

I had thought that to mean I was specifying up to 5 GPU's (whereas I'm only assigned / using 2 GPU's in this example). Calling deepspeed as above resulted in the node prompting me for its password, which of course was also not desired behavior.

Expected behavior
I expected deepspeed to inherit the specific GPU numeric assignments from CUDA_VISIBLE_DEVICES. It seems as though Deepspeed always re-indexes the assignments of the GPU's to start from 0. I believe this code snippet shows how the values given to the world_info dictionary are created (specifically line 306: list(range(args.num_gpus))).

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
async_io ............... [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/torch']
torch version .................... 1.9.0
torch cuda version ............... 10.2
nvcc version ..................... 10.2
deepspeed install path ........... ['/envs/huggingface_deepspeed_python/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.5.0, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.9, cuda 10.2

Software Version Notes
I'm running Python 3.9.6 within a Conda library where I've installed deepspeed via pip (and all other libraries via conda install).

Additional Notes and Future Usage
Currently I'm only running these workflows on single node, which has anywhere from 8 to 16 GPU's. However, I am interested in applying these workflows across multiple nodes that have high-speed interconnects.

Please let me know if you require any further information.

Start in deepspeed/launcher/runner.py, especially the resource parsing and world_info construction around the referenced lines 270–306, then trace how deepspeed.launcher.launch applies the selected devices. Reproduce the shown SLURM CUDA_VISIBLE_DEVICES and --include cases, including the hostfile parser. Done means the launcher preserves assigned GPU identifiers without selecting unavailable or already-used devices, with the demonstrated single-node commands behaving correctly.