[Feature] Support autoscaling of MiG-enabled GPU nodes
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
Nvidia introduced a feature called [MiG](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html) wherein it allows a GPU node to be partitioned into multiple isolated instances. This allows for scheduling multiple pods per GPU node - a thing that wasn't possible before.
This allows for better scaling and potentially more efficient utilization of GPUs.
With nvidia-device-plugin and gpu-operator you can now target pods on these isolated MiG partitions by specifying it in the resource requests and limits for ex: `nvidia.com/mig-1g.5gb: 1` (which previously was `nvidia.com/gpu: 1`)
While MiG partitioning is supported by AKS, but autoscaling of MiG-enabled GPU nodepool isn't. This is because the resource `nvidia.com/gpu` is [hardcoded](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/utils/gpu/gpu.go#L28) on the cluster autoscaler. There is already an issue raised on that repo here: [Nvidia resource name is hardcoded to nvidia.com/gpu · Issue #8095 · kubernetes/autoscaler](https://github.com/kubernetes/autoscaler/issues/8095)
After talking to Azure Support, I was asked to raise the issue here as well.
Optional: Set the appropriate GitHub label(s) to indicate the AKS feature or area your feature request relates to.
**Describe the solution you'd like**
When gpu nodepools are deployed with mig enabled, these should schedule workloads and autoscale on demand just like any other node. Today that doesn't happen.
**Describe alternatives you've considered**
One could potentially use [MachineDeployments](https://github.com/kubernetes/autoscaler/blob/ce81a6a43c6617475f219d1036d761715edf45ec/cluster-autoscaler/cloudprovider/clusterapi/README.md?plain=1#L222) but I haven't explored that path further.
Contributor guide
Assessment
This issue has not been assessed yet.