aws / aws/aws-parallelcluster

gres and gpus-per-node only lists t4 when g4 (T4) and g5 (A10G) are set

Open
#5,880 3 comments 0 reactions 0 assignees View on GitHub
3.x Feature Request
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

- AWS ParallelCluster version [e.g. 3.1.1]: 3.7.0

Configured a cluster with a GPU queue with 2 instance types
InstanceType: g4dn.2xlarge
InstanceType: g5.2xlarge

srun only allows type t4 for gpu type, not a10g
`srun -p gpu --time=6:00:00 -c 8 --gpus-per-node=1 --gpus-per-node=t4:1 --pty /bin/bash -i` <--- Works
`srun -p gpu --time=6:00:00 -c 8 --gpus-per-node=1 --gpus-per-node=a10g:1 --pty /bin/bash -i` <--- Doesn't

Files
/opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_partition.conf
/opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf

Only contains the g4 instances, no g5
```
$ cat /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf
# This file is automatically generated by pcluster
NodeName=gpu-dy-g4dn2xlarge-[1-128] Name=gpu Type=t4 File=/dev/nvidia[0-0]
$
$ cat /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_partition.conf
# This file is automatically generated by pcluster

NodeName=gpu-dy-g4dn2xlarge-[1-128] CPUs=8 RealMemory=31129 State=CLOUD Feature=dynamic,g4dn2xlarge,gpu Weight=2000 Gres=gpu:t4:1

NodeSet=gpu_nodes Nodes=gpu-dy-g4dn2xlarge-[1-128]
PartitionName=gpu Nodes=gpu_nodes MaxTime=INFINITE State=UP
$
```

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the generated /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf and slurm_parallelcluster_gpu_partition.conf for a cluster configured with g4dn.2xlarge and g5.2xlarge. Reproduce the two srun commands and trace why only t4 is listed; done means the generated files include the g5/A10G GPU and --gpus-per-node=a10g:1 works.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.