NVIDIA / NVIDIA/k8s-device-plugin
[Bug]: Health check misses devices that share a physical GPU
Open
@aryangorwade is already working on this.
Since Sep 11, 2026.
bug
needs-triage
- Dominant language
- Go
- Stars
- 3.9k
- Forks
- 868
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 27
Description
1. Quick Debug Information
- OS/Version(e.g. RHEL8.6, Ubuntu22.04): Amazon Linux 2023
- Kernel Version: Kernel-6.1
- Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd
- K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): EKS(v1.34)
2. Issue or feature description
- Without MIG (time slicing or MPS). When multiple logical GPUs (replicas) correspond to one physical GPU UUID, an Xid error on that UUID should report all of them as unhealthy to the kubelet. Currently only one logical GPU is reported. Since the physical GPU itself is broken, every logical GPU tied to it should become unhealthy.
- With MIG. Even when multiple MIG instances inside one physical GPU are broken, at most one is reported as unhealthy, and in some cases none at all. If the value of
parentToDeviceMap[uuid]happens to be a pointer to a device that is not broken, the GI and CI check rejects the event and nothing is reported.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.