[Bug]: Crashlooping nvidia-peermem-ctr: could not insert 'nvidia_peermem': Invalid argument

Describe the bug

Filing a bug after the conversation at https://github.com/NVIDIA/gpu-operator/issues/2844 thanks

On a GB300 on-prem cluster we're seeing constant crashloops of nvidia-peermem-ctr:

DRIVER_ARCH is aarch64
modprobe: ERROR: could not insert 'nvidia_peermem': Invalid argument
failed to load nvidia-peermem module

Recent dmesg -T | grep -i -E 'peermem|ib_core|peer_mem':

[Tue Sep  8 17:16:02 2026] nvidia_peermem: disagrees about version of symbol ib_register_peer_memory_client
[Tue Sep  8 17:16:02 2026] nvidia_peermem: Unknown symbol ib_register_peer_memory_client (err -22)
[Tue Sep  8 17:16:02 2026] nvidia_peermem: disagrees about version of symbol ib_unregister_peer_memory_client
[Tue Sep  8 17:16:02 2026] nvidia_peermem: Unknown symbol ib_unregister_peer_memory_client (err -22)
[Tue Sep  8 17:21:18 2026] nvidia_peermem: disagrees about version of symbol ib_register_peer_memory_client
[Tue Sep  8 17:21:18 2026] nvidia_peermem: Unknown symbol ib_register_peer_memory_client (err -22)
[Tue Sep  8 17:21:18 2026] nvidia_peermem: disagrees about version of symbol ib_unregister_peer_memory_client
[Tue Sep  8 17:21:18 2026] nvidia_peermem: Unknown symbol ib_unregister_peer_memory_client (err -22)

lsmod | grep -E 'nvidia|mlx5|ib_':

mlx5_dpll             196608  0
mlx5_vdpa             196608  0
ib_ipoib              262144  0
ib_cm                 327680  2 rdma_cm,ib_ipoib
ib_umad               262144  0
mlx5_fwctl            262144  0
fwctl                 262144  1 mlx5_fwctl
mlx5_ib               655360  0
ib_uverbs             327680  2 rdma_ucm,mlx5_ib
mlx5_core            3145728  3 mlx5_dpll,mlx5_fwctl,mlx5_ib
mlxfw                 262144  1 mlx5_core
mlxdevm               655360  1 mlx5_core
ib_core               720896  8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx_compat            196608  15 mlx5_dpll,rdma_cm,ib_ipoib,mlxdevm,mlx5_fwctl,iw_cm,ib_umad,mlx5_vdpa,fwctl,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
nvidia_modeset       2097152  0
nvidia_uvm           1966080  8
nvidia              15138816  55 nvidia_uvm,gdrdrv,nvidia_modeset
video                 262144  1 nvidia_modeset
ecc                   196608  1 nvidia
nvidia_cspmu          196608  0
arm_cspmu_module      262144  1 nvidia_cspmu
macsec                262144  1 mlx5_ib
psample               262144  1 mlx5_core
tls                   327680  1 mlx5_core
pci_hyperv_intf       196608  1 mlx5_core

To Reproduce

No clear way to get to those errors. Half of the nodes seem to load the module, half do not. No reasonable split of configs / kernels / ... so far explained this split.

Expected behavior

Not crash :-)

Environment (please provide the following information):

• GPU Operator Version: v26.7.0
• OS: Ubuntu24.04
• Kernel Version: 6.8.0-124-generic-64k or 6.8.0-136-generic-64k
• Container Runtime Version: containerd 2.2.3
• Kubernetes Distro and Version: k8s v1.35.3

DOCA: doca3.5.0-26.07-0.7.7.0-0-ubuntu24.04-arm64

@rajathagasthya LMK if a full debug bundle would be helpful

@lalitadithya FWIW: in this case nvsentinel is not actually picking up any errors, interestingly. The only "signal" from an administrator perspective is that pods are crashlooping continuously (which perhaps should be enough, but ... do consider if detection here would make sense thanks).