[Feature]: Allow overriding the derived OS-tag suffix on driver image versions - OKD/SCOS nodes (os_release ID=centos) resolve to nonexistent -centos10 tags
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
In OKD deployments (okd-project/okd) the nodes run CentOS Stream CoreOS (SCOS), whose NFD labels report feature.node.kubernetes.io/system-os_release.ID=centos and …VERSION_ID=10. getOSTag() in internal/state/nodepool.go therefore renders the driver image suffix as centos10 → nvcr.io/nvidia/driver:<version>-centos10, a tag that is not published, and the driver pod fails with ImagePullBackOff. RHEL and its clones are already special-cased there (ol, rocky, rhel → major-only suffix); centos falls through to the default branch.
The documented workaround (confirmed in #1533) is pinning spec.driver.version to a digest, which is used verbatim. That works, but freezes the image generation: rebuilds never flow under the same pin, leaving no tag-based upgrade path for the pinned reference. In practice we also hit the pinned tag being removed from nvcr.io (gpu-driver-container#985), and the newer images we would otherwise move to were blocked by a separate issue (gpu-driver-container#984).
Expected behavior: users running OKD/SCOS should be able to select a published, compatible driver image suffix (or otherwise prevent the operator from constructing an unavailable -centos10 tag).
One shape that would solve it: a CRD-level override of the derived OS tag — e.g. spec.driver.osTag: rhel10.0 — taking precedence over the os_release-derived suffix. A smaller alternative in the same spirit: extending getOSTag's existing ol/rocky/rhel normalization to map centos → rhel. For context on compatibility: OKD is the community OpenShift distribution and the operator's OpenShift/DTK integration is active on these nodes (not a standalone CentOS host), and we build against the published -rhel10 driver images on SCOS today (digest-pinned 580.126.20-rhel10.0, kernel 6.12.0-233.el10) with the kmod building and loading fine. Happy with either shape — whichever better fits the project's support model.
Related: #1533 (original ask, resolved via the digest workaround, closed stale) · #2811 (downstream: repoConfig in DTK container) · gpu-driver-container#984 / #985
Environment: GPU Operator v26.3.3 · OKD 4.22.0-okd-scos.6 · CentOS Stream CoreOS 10, kernel 6.12.0-233.el10.x86_64 · NVIDIA L4 · amd64
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/state/nodepool.go at getOSTag and trace how spec.driver.version and the driver CRD are represented. Compare the existing ol/rocky/rhel normalization with the proposed CRD-level override, then confirm done means OKD/SCOS resolves to a published compatible suffix without constructing -centos10 tags.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100