[Bug]: `nvidia-operator-validator` reports Ready while driver device nodes (`nvidia-uvm`/`nvidia-uvm-tools`) are missing — false positive forces users to run their own CUDA smoke test

Describe the bug

nvidia-operator-validator can become Ready (all init validations pass) while the driver is not
actually usable on the host: after a node reboot or a recreation of the nvidia-driver-daemonset
pod (the driver DaemonSet uses a hard-coded OnDelete update strategy, so pod recreation is a
routine lifecycle event), the device nodes under the driver root — notably
/run/nvidia/driver/dev/nvidia-uvm and /run/nvidia/driver/dev/nvidia-uvm-tools — are sometimes
missing or stale. The validator still reports success, nvidia.com/gpu capacity is advertised, and
pods get scheduled, but any real CUDA workload fails to initialize (NVML/UVM errors, or the pod
hangs waiting on the device node).

In other words, the validator checks that the driver container came up and nvidia-smi works, but
it does not verify that the full set of device nodes required by CUDA workloads exists and works,
and it does not re-validate when the driver pod is recreated while the node keeps running.

Because of this false positive we cannot trust nvidia-operator-validator as the readiness signal
for our fleet of unattended edge devices. We had to build our own validation for the self-healing
process: a CUDA vectoradd smoke-test Job whose init container waits for
/run/nvidia/driver/dev/nvidia-uvm-tools, plus a boot-time watchdog that deletes/recreates the
nvidia-driver-daemonset pod whenever the smoke test stalls (recreating the driver pod is the
action that reliably refreshes the device nodes and unblocks the workload). This works, but it
duplicates what the operator-validator is supposed to provide.

Related reports that touch parts of this gap:

• #475 — validation completes after workloads already scheduled; no reliable "validated" signal
• #2166 — driver pod restart breaks running GPU workloads
• #539 — validator fails to create host device symlinks
• #1923 — nvidia-uvm module load failure surfaced only indirectly

To Reproduce

Environment: unattended edge node (immutable Ubuntu 24.04 image), RKE2, GPU Operator with a
precompiled/signed driver container keyed to the host kernel, single NVIDIA RTX GPU shared via
time-slicing (replicas: 5).

1. Install the GPU Operator (Helm) with driver.enabled: true and a precompiled driver image,
   plus time-slicing device-plugin config.
2. Reboot the node (or delete the nvidia-driver-daemonset pod so the OnDelete strategy
   recreates it while the node keeps running).
3. Wait for nvidia-operator-validator to become Ready — all validator init containers
   (driver-validation, toolkit-validation, cuda-validation, plugin-validation) succeed.
4. Check the driver root on the host: ls /run/nvidia/driver/dev/ | grep uvm — intermittently
   nvidia-uvm-tools (and/or nvidia-uvm) is missing.
5. Schedule any CUDA pod (e.g. nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0 with
   nvidia.com/gpu: 1): it fails to initialize CUDA/UVM, while the validator continues to report
   the node as validated.
6. Delete the nvidia-driver-daemonset pod; after it is recreated, the device nodes appear and the
   same workload succeeds — confirming the validator signaled Ready on a non-working driver state.

The race is intermittent (boot-ordering dependent); across a fleet of hundreds of devices it occurs
on a meaningful fraction of boots.

Expected behavior

• nvidia-operator-validator should only report Ready when the driver is actually usable by a
  workload: all required device nodes present under the driver root (nvidia, nvidiactl,
  nvidia-modeset, nvidia-uvm, nvidia-uvm-tools, ...) and a real CUDA context can be created —
  i.e. the validation should catch the state that our external vectoradd smoke test catches.
• Validation should be re-triggered whenever the driver DaemonSet pod restarts/recreates, instead
  of remaining green from a previous validation cycle.
• Ideally the validation result would be exposed as a trustworthy node signal (label/taint/
  condition, cf. #475) so cluster automation can gate GPU workloads on it without running a
  parallel smoke-test pipeline.

Environment (please provide the following information):

• GPU Operator Version: v26.3.2 (Helm chart, NVIDIA registry)
• OS: Ubuntu 24.04 (immutable/A-B image-based edge OS, Kairos-derived)
• Kernel Version: 6.8.0-101-generic
• Container Runtime Version: containerd (embedded in RKE2)
• Kubernetes Distro and Version: RKE2 (edge cluster managed via Spectro Cloud Palette)

Additional configuration details:

• Driver: precompiled/signed driver container, version 580.126.09, built per kernel version, served
  from a private registry mirror
• k8s-driver-manager: v0.11.0, container-toolkit: v1.19.1, device-plugin: v0.19.2
• Device plugin time-slicing: nvidia.com/gpu replicas: 5 on a single RTX GPU
• Single-node clusters, unattended operation (no operator at the keyboard — self-healing must be
  automatic)

Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods -n gpu-operator — all pods Running/Completed,
  nvidia-operator-validator 1/1 Ready in the failing state
• kubernetes daemonset status: kubectl get ds -n gpu-operator — desired == ready for all
  daemonsets in the failing state
• kubectl describe pod — the validator is not in an error state (that is the bug); our
  smoke-test pod is stuck Init:0/1 waiting for /run/nvidia/driver/dev/nvidia-uvm-tools
• kubectl logs of validator containers — all validations report success
• nvidia-smi from the driver container succeeds even while nvidia-uvm-tools is missing —
  which is why nvidia-smi-based validation does not catch this
• containerd logs — no runtime-level errors; failure only surfaces at CUDA/UVM init inside the
  workload

Workaround we currently run (what the validator should make unnecessary) — smoke-test Job deployed
next to the operator, watched by a host-side heal daemon that recreates the driver pod if the job
stalls:

apiVersion: batch/v1
kind: Job
metadata:
  name: nvidia-gpu-smoke-test
  namespace: gpu-operator
  labels:
    app: nvidia-gpu-smoke-test
spec:
  parallelism: 1
  completions: 1
  backoffLimit: 50
  template:
    metadata:
      labels:
        app: nvidia-gpu-smoke-test
    spec:
      runtimeClassName: nvidia
      restartPolicy: OnFailure
      initContainers:
      - name: wait-driver
        image: busybox:1.36
        command:
        - sh
        - -c
        - |
          echo "waiting for driver"
          until [ -e /run/nvidia/driver/dev/nvidia-uvm-tools ]; do
            sleep 2
          done
        volumeMounts:
        - mountPath: /run/nvidia
          name: nvidia-run
          readOnly: true
      containers:
      - name: test
        image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
        resources:
          limits:
            nvidia.com/gpu: 1
      volumes:
      - name: nvidia-run
        hostPath:
          path: /run/nvidia

Heal logic (pseudocode of the host watchdog):

wait for kube-apiserver, then for deploy/gpu-operator Available
loop until job/nvidia-gpu-smoke-test is Complete:
  recreate smoke-test job + delete driver pods (graceful)
  wait driver DS Ready      (on stall: force-delete driver pods on ImagePull/CrashLoop)
  wait operator-validator Ready (on stall: delete validator pods)
  wait smoke test Complete  (if its init container reports the uvm-tools device is missing:
                             delete the driver pod → device nodes are refreshed → test passes)

References:

• Driver upgrades / OnDelete strategy: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html
• Time-slicing: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html