Azure / Azure/AKS

[Feature] Support autoscaling of MiG-enabled GPU nodes

Open
#5,161 6 comments 2 reactions 2 assignees Claimed by @kevinkrp93 View on GitHub
cluster-autoscaler feature-request GPU upstream
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**

Nvidia introduced a feature called [MiG](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html) wherein it allows a GPU node to be partitioned into multiple isolated instances. This allows for scheduling multiple pods per GPU node - a thing that wasn't possible before.
This allows for better scaling and potentially more efficient utilization of GPUs.

With nvidia-device-plugin and gpu-operator you can now target pods on these isolated MiG partitions by specifying it in the resource requests and limits for ex: `nvidia.com/mig-1g.5gb: 1` (which previously was `nvidia.com/gpu: 1`)

While MiG partitioning is supported by AKS, but autoscaling of MiG-enabled GPU nodepool isn't. This is because the resource `nvidia.com/gpu` is [hardcoded](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/utils/gpu/gpu.go#L28) on the cluster autoscaler. There is already an issue raised on that repo here: [Nvidia resource name is hardcoded to nvidia.com/gpu · Issue #8095 · kubernetes/autoscaler](https://github.com/kubernetes/autoscaler/issues/8095)

After talking to Azure Support, I was asked to raise the issue here as well.

Optional: Set the appropriate GitHub label(s) to indicate the AKS feature or area your feature request relates to.

**Describe the solution you'd like**

When gpu nodepools are deployed with mig enabled, these should schedule workloads and autoscale on demand just like any other node. Today that doesn't happen.

**Describe alternatives you've considered**

One could potentially use [MachineDeployments](https://github.com/kubernetes/autoscaler/blob/ce81a6a43c6617475f219d1036d761715edf45ec/cluster-autoscaler/cloudprovider/clusterapi/README.md?plain=1#L222) but I haven't explored that path further.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.