nvidia-peermem sidecar crashloops after "userspace-only install", wedging RDMA nodes and the driver upgrade flow

Environment

• gpu-operator v26.3.2, driver 580.126.20 (open kernel modules), container-toolkit v1.19.1
• AKS, Kubernetes 1.35.5, Ubuntu 24.04, kernel 6.8.0-1059-azure, containerd 2.3.2
• Standard_ND96isr_H100_v5 nodes, driver.rdma.enabled=true (MOFED via network-operator DOCA driver container, useHostMofed=false)

Symptom

After a driver daemonset pod restarts on a node where the driver is already loaded, the pod sticks at 2/3 indefinitely:

nvidia-driver-ctr:   ready=true   (log: "performing userspace-only install")
nvidia-gdrcopy-ctr:  ready=true
nvidia-peermem-ctr:  CrashLoopBackOff
  modprobe: FATAL: Module nvidia-peermem not found in directory /lib/modules/6.8.0-1059-azure

The daemonset never goes Ready, so the driver-upgrade state machine stalls (node stays cordoned in pod-restart-required / validation-required) and ClusterPolicy stays notReady.

Root cause

When the driver container starts and finds the desired driver version already loaded, it takes the reuse path: "The NVIDIA driver is already loaded with the desired configuration, performing userspace-only install" (nvidia-installer runs with --no-kernel-modules). That path never populates /lib/modules/<kernel> with the kmod tree in the new pod's mount namespace, and the container's startup cleanup removes any tree left by the previous pod. The nvidia-peermem sidecar then runs a disk-based modprobe, which fails — even though the module is loaded and fully functional in the kernel:

# lsmod (on the same node, at the same moment)
nvidia_peermem   16384  0
ib_uverbs        196608 67  nvidia_peermem,rdma_ucm,mlx5_ib
# find /lib/modules -name 'nvidia-peermem*'  ->  (nothing)

Restarting the pod can never recover: every restart re-detects the loaded driver and repeats the userspace-only install. The sidecar's check conflates "module file on disk" with "module available", and the reuse path breaks that assumption.

Two trigger paths observed (same day, same cluster)

1. Manual driver pod restart (kubectl delete pod) with modules left loaded.
2. The operator's own upgrade-controller roll (pod-restart-required): k8s-driver-manager's unload failed silently because a crashlooping dcgm-exporter pod still held /dev/nvidia-uvm handles; the flow proceeded anyway, landing the new pod on the reuse path. So this is reachable inside the fully managed path, not just via manual intervention.

Verification (retained evidence, post-recovery)

• Affected driver pod showed nvidia-driver-ctr restartCount=1 and nvidia-peermem-ctr restartCount=2; previous-container logs retain both the "userspace-only install" line (with the --no-kernel-modules warning) and the peermem load failure.
• Cluster events retain the peermem BackOff during the GPUDriverUpgrade state flow, plus dcgm-exporter restart/sandbox failures while the node cycled through pod-restart/validation.
• After recovery via full reinstall, the same nodes are healthy: driver pods 3/3, ClusterPolicy ready, nvidia_peermem loaded, and module files present under /lib/modules and /run/nvidia/driver/lib/modules.

Workaround (validated)

Per affected node, hold the daemonset off while unloading, then let it reinstall fresh:

kubectl label node <node> nvidia.com/gpu.deploy.driver=false nvidia.com/gpu.deploy.dcgm-exporter=false --overwrite
# wait for the driver + exporter pods to fully terminate (they hold /dev/nvidia*)
rmmod nvidia_peermem gdrdrv nvidia_uvm nvidia_modeset nvidia    # on the node; retry if "in use"
kubectl label node <node> nvidia.com/gpu.deploy.driver=true nvidia.com/gpu.deploy.dcgm-exporter=true --overwrite

The label flip matters: with plain pod deletion, the recreated pod races the unload, re-detects the loaded driver, and re-acquires the devices. A node reboot is an equivalent, blunter alternative.

Suggested fixes

1. The peermem sidecar should treat "module already loaded" as success (check /sys/module or lsmod before the disk modprobe), and/or
2. The userspace-only install path should still lay down the kernel-module tree so disk-based modprobe stays valid, and
3. k8s-driver-manager should fail hard (retry/backoff) when module unload fails, instead of proceeding into a state the sidecars can't handle.