[Feature]: Ignore already cordoned nodes during driver upgrades
@JunAr7112 is already working on this.
Since Sep 10, 2026.
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
Requestor: @kvalliyurnatt
Summary
Ignore nodes that are already cordoned (not by any of the GPU operator components) during upgrades from upgrade controller/k8s-driver-manager
Motivation
If there is a node that has already been cordoned (not by any of the GPU operator components), then that means the node is not accepting any workloads at the moment, and there is no guarantee of the state that node is in, it could be down or in an unresponsive state. So it feels like the upgrade controller/k8s-driver-manager should ignore nodes that are already cordoned and not take any action on such nodes until they are uncordoned.
Proposal
We will need a way differentiate nodes that are cordoned by the GPU operator components, for which we could add an annotation on the nodes when the upgrade controller/k8s-driver-manager cordons a node(I believe the upgrade controller already does this) and then uncordon nodes based on that annotation. Then the upgrade controller/k8s-driver-manager can ignore nodes that are in a cordoned state but don't have the annotation on them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.