gres and gpus-per-node only lists t4 when g4 (T4) and g5 (A10G) are set
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
- AWS ParallelCluster version [e.g. 3.1.1]: 3.7.0
Configured a cluster with a GPU queue with 2 instance types
InstanceType: g4dn.2xlarge
InstanceType: g5.2xlarge
srun only allows type t4 for gpu type, not a10g
`srun -p gpu --time=6:00:00 -c 8 --gpus-per-node=1 --gpus-per-node=t4:1 --pty /bin/bash -i` <--- Works
`srun -p gpu --time=6:00:00 -c 8 --gpus-per-node=1 --gpus-per-node=a10g:1 --pty /bin/bash -i` <--- Doesn't
Files
/opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_partition.conf
/opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf
Only contains the g4 instances, no g5
```
$ cat /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf
# This file is automatically generated by pcluster
NodeName=gpu-dy-g4dn2xlarge-[1-128] Name=gpu Type=t4 File=/dev/nvidia[0-0]
$
$ cat /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_partition.conf
# This file is automatically generated by pcluster
NodeName=gpu-dy-g4dn2xlarge-[1-128] CPUs=8 RealMemory=31129 State=CLOUD Feature=dynamic,g4dn2xlarge,gpu Weight=2000 Gres=gpu:t4:1
NodeSet=gpu_nodes Nodes=gpu-dy-g4dn2xlarge-[1-128]
PartitionName=gpu Nodes=gpu_nodes MaxTime=INFINITE State=UP
$
```
Contributor guide
Research direction
Start by inspecting the generated /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf and slurm_parallelcluster_gpu_partition.conf for a cluster configured with g4dn.2xlarge and g5.2xlarge. Reproduce the two srun commands and trace why only t4 is listed; done means the generated files include the g5/A10G GPU and --gpus-per-node=a10g:1 works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100