no slot '4' specified on local host - trying to use 4 gpus on a node with 8 gpus while another user is using the other 4 gpus

I am running a DeepSpeed training job on SLURM, requesting 4 GPUs on a shared node. SLURM correctly assigns my job GPUs 4,5,6,7, but DeepSpeed remaps them to 0,1,2,3, causing conflicts with another user’s job.

Error Message:

ValueError: No slot '4' specified on host 'localhost'

DeepSpeed misinterprets the physical GPU IDs as local device indices, leading to an out-of-range error. nvidia-smi confirms that my job is running on the wrong GPUs (0-3) instead of the assigned GPUs (4-7), causing resource conflicts.

SLURM Script Configuration:

#SBATCH --gres=gpu:h100:4
#SBATCH --mem=300G
#SBATCH --cpus-per-task=48

Before launching DeepSpeed, I set:

export CUDA_VISIBLE_DEVICES="$SLURM_JOB_GPUS"
echo "CUDA_VISIBLE_DEVICES: $CUDA_VISIBLE_DEVICES"

However, DeepSpeed still ignores this setting and remaps GPUs.

Troubleshooting Attempts:

1. Remapping SLURM_JOB_GPUS to a Zero‐Indexed List

   export CUDA_VISIBLE_DEVICES=$(seq -s, 0 $((num_gpus - 1)))
   

   ❌ Failure: DeepSpeed still assigned GPUs 0,1,2,3, conflicting with another user's job.

2. Setting CUDA_VISIBLE_DEVICES Directly to SLURM_JOB_GPUS

   export CUDA_VISIBLE_DEVICES="$SLURM_JOB_GPUS"
   

   ❌ Failure: DeepSpeed detected CUDA_VISIBLE_DEVICES=4,5,6,7 but ignored it, using --include=localhost:0,1,2,3 instead.

3. Explicitly Passing a Zero‐Indexed --include Flag

   deepspeed --include=localhost:0,1,2,3
   

   ❌ Failure: DeepSpeed still reassigned CUDA_VISIBLE_DEVICES=0,1,2,3.

4. Unsetting SLURM_JOB_GPUS and Removing --include

   unset SLURM_JOB_GPUS
   deepspeed --master_port $MASTER_PORT train.py --deepspeed ./scripts/config.json
   

   ❌ Failure: DeepSpeed again reassigned CUDA_VISIBLE_DEVICES=0,1,2,3.

Final Observations:

• DeepSpeed Overrides CUDA_VISIBLE_DEVICES
  Even when explicitly set, DeepSpeed overrides CUDA_VISIBLE_DEVICES if any of the following flags are used:
  --include, --exclude, --num_gpus, --num_nodes.

• DeepSpeed Reassigns GPUs Internally
  DeepSpeed assumes GPUs should always be indexed 0,1,2,3, regardless of SLURM's physical GPU assignment, causing a mismatch.

• nvidia-smi Confirms GPU Conflict
  My job (psy_llava) is running on the same GPUs as another job (python), despite SLURM allocating different GPUs.

• No Known Fix Has Worked
  Every attempted fix (including removing --include) has failed to prevent DeepSpeed from overriding CUDA_VISIBLE_DEVICES.
  This suggests a deeper issue with DeepSpeed's GPU initialization in SLURM-managed environments.

Seeking Help:

• How can I force DeepSpeed to respect SLURM’s GPU allocation and prevent it from overriding CUDA_VISIBLE_DEVICES?
• Are there any known DeepSpeed settings to prevent it from remapping GPUs?
• Is there a recommended way to ensure that DeepSpeed only runs on the GPUs explicitly assigned by SLURM?

Start from the deepspeed launcher invocation and reproduce the SLURM setup using SLURM_JOB_GPUS, CUDA_VISIBLE_DEVICES, and --include. Trace how the launcher selects local devices and compare that with the reported No slot '4' error. Done means the job uses only SLURM-assigned GPUs without remapping or conflicts.