[Bug]: High CPU Usage on Nontrivial MIG Configurations

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): Talos 1.13.0
• Kernel Version: 6.18.24-talos
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): containerd://2.2.3
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): K8s 1.35.5

2. Issue or feature description

We're running v26.3.3 of the GPU operator chart on Talos VMs with one H100 GPU each, attached via PCI passthrough.

The device plugin pods use 100% CPU on nodes with MIG layouts with multiple slices. On nodes where MIG is disabled or set to a single-slice layout (in our case, a single 7g.94gb partition), the CPU usage is minimal. CPU usage rises to 100% pretty much immediately after the pod starts. Known affected layouts:

• 1x 4g.47gb and 1x 3g.47gb
• 1x 3g.47gb and 2x 2g.24gb

Known unaffected layouts:

• 1x 7g.94gb
• MIG disabled, i.e. mig-enabled: false in the mig-parted configuration

The same behavior was described in #1542, but the issue was closed since OP did not respond to comments.

3. Information to attach (optional if deemed irrelevant)

From the Talos dashboard:

The logs don't show any particular errors. The pods with low and high CPU usage have basically the same output, aside from the resources advertised. The error ERROR: init 250 result=11 appears in all the NVIDIA pods but doesn't seem to affect anything. I've attached the full logs below. The pod starts the GRPC servers for the configured MIG slices and then nothing else gets printed. The device plugin responds correctly to changes to the nvidia.com/device-plugin.config label.

Note: the MIG manager doesn't work on Talos. See https://github.com/NVIDIA/mig-parted/pull/382, which adds support for Talos. We are using this build.

Common error checking:

• The output of nvidia-smi -a on your host
• Your docker configuration file (e.g: /etc/docker/daemon.json)
• The k8s-device-plugin container logs
  • device-plugin.log
• The kubelet logs on the node (e.g: sudo journalctl -r -u kubelet)

Additional information that might help better understand your environment and reproduce the bug:

• Docker version from docker version
• Docker command, image and tag used
• Kernel version from uname -a
• Any relevant kernel output lines from dmesg
• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'
  • Installed via Talos system extensions
  • extensions.talos.dev/nonfree-kmod-nvidia-lts=580.126.20-v1.13.0
  • extensions.talos.dev/nvidia-container-toolkit-lts=580.126.20-v1.19.0
• NVIDIA Container Toolkit version from nvidia-ctk --version