NVIDIA / NVIDIA/gpu-operator

[Feature]: Allow overriding the derived OS-tag suffix on driver image versions - OKD/SCOS nodes (os_release ID=centos) resolve to nonexistent -centos10 tags

Open
#2,892 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.9k
Forks
552
Avg merge
2d 4h
Merged PRs (30d)
90

Description

In OKD deployments (okd-project/okd) the nodes run CentOS Stream CoreOS (SCOS), whose NFD labels report feature.node.kubernetes.io/system-os_release.ID=centos and …VERSION_ID=10. getOSTag() in internal/state/nodepool.go therefore renders the driver image suffix as centos10nvcr.io/nvidia/driver:<version>-centos10, a tag that is not published, and the driver pod fails with ImagePullBackOff. RHEL and its clones are already special-cased there (ol, rocky, rhel → major-only suffix); centos falls through to the default branch.

The documented workaround (confirmed in #1533) is pinning spec.driver.version to a digest, which is used verbatim. That works, but freezes the image generation: rebuilds never flow under the same pin, leaving no tag-based upgrade path for the pinned reference. In practice we also hit the pinned tag being removed from nvcr.io (gpu-driver-container#985), and the newer images we would otherwise move to were blocked by a separate issue (gpu-driver-container#984).

Expected behavior: users running OKD/SCOS should be able to select a published, compatible driver image suffix (or otherwise prevent the operator from constructing an unavailable -centos10 tag).

One shape that would solve it: a CRD-level override of the derived OS tag — e.g. spec.driver.osTag: rhel10.0 — taking precedence over the os_release-derived suffix. A smaller alternative in the same spirit: extending getOSTag's existing ol/rocky/rhel normalization to map centosrhel. For context on compatibility: OKD is the community OpenShift distribution and the operator's OpenShift/DTK integration is active on these nodes (not a standalone CentOS host), and we build against the published -rhel10 driver images on SCOS today (digest-pinned 580.126.20-rhel10.0, kernel 6.12.0-233.el10) with the kmod building and loading fine. Happy with either shape — whichever better fits the project's support model.

Related: #1533 (original ask, resolved via the digest workaround, closed stale) · #2811 (downstream: repoConfig in DTK container) · gpu-driver-container#984 / #985

Environment: GPU Operator v26.3.3 · OKD 4.22.0-okd-scos.6 · CentOS Stream CoreOS 10, kernel 6.12.0-233.el10.x86_64 · NVIDIA L4 · amd64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/state/nodepool.go at getOSTag and trace how spec.driver.version and the driver CRD are represented. Compare the existing ol/rocky/rhel normalization with the proposed CRD-level override, then confirm done means OKD/SCOS resolves to a published compatible suffix without constructing -centos10 tags.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.