NVIDIA / NVIDIA/cloud-native-stack
pod needing GPU or SRIOV failing scheduling after node restart
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 249
- Forks
- 85
- PR merge metrics
- No merged PRs in 30d
Description
After node restart some pods in needs of GPU or VF might fails because the device ressources are not yet ready.
It requires manual delete of failing pods.
as example, one error can be reported by the failing pod as:
Message: Pod was rejected: Allocate failed due to no healthy devices present; cannot allocate unhealthy devices openshift.io/media_a_rx_pool, which is unexpected,
As one solution, we could implement a daemonset that deletes those pods once all gpu/vf nodes have at least 1 device allocatable. I've done this
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the Kubernetes device-readiness behavior described in the issue and the proposed DaemonSet approach. Done means GPU- or SR-IOV-dependent pods recover after a node restart once devices are allocatable, without manual pod deletion, and the reported unhealthy-device failure path is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100