[BUG] nvidia-device-plugin fails with NVIDIA drivers due to missing versioned libnvidia-gpucomp.so in WSL mount
- Dominant language
- PowerShell
- Stars
- 66
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
# `nvidia-device-plugin` fails with NVIDIA drivers >= 595.x due to missing versioned `libnvidia-gpucomp.so` in WSL mount
## Environment
- Windows 11 25H2 (build 26200.7840)
- AKS-EE 1.11.247.0 (Linux node)
- NVIDIA driver 595.79 (595.54 internal)
- nvidia-device-plugin 0.18.2
## Problem
The `nvidia-device-plugin` daemonset pod fails to start with:
```
nvidia-container-cli: mount error: lstat failed:
/var/.eflow/custom-configs/usr/lib/wsl/lib/libnvidia-gpucomp.so.595.54:
no such file or directory
```
## Root cause
NVIDIA drivers >= 595.x introduce `libnvidia-gpucomp.so` in the WSL library mount (`C:\Windows\System32\lxss\lib`). This file is exposed in the AKS-EE Linux node at `/usr/lib/wsl/lib/libnvidia-gpucomp.so` — **without** a version number in the filename. However, the ELF SONAME embedded in the binary is `libnvidia-gpucomp.so.595.54` — **with** a driver-specific version number.
When `ldconfig` scans `/usr/lib/wsl/lib` at boot, it registers the library as:
```
libnvidia-gpucomp.so.595.54 => /usr/lib/wsl/lib/libnvidia-gpucomp.so.595.54
```
The nvidia-container-runtime then attempts to bind-mount that versioned path into the container — but that file does not exist. All other libraries in `lxss\lib` (e.g. `libnvidia-ml.so.1`, `libnvidia-encode.so.1`) have a **fixed** version suffix in their filename that matches their SONAME, so they do not have this problem.
This worked correctly with NVIDIA driver 572.83, which does not include `libnvidia-gpucomp.so` at all.
## Workaround
Copy the library with the versioned filename into `lxss\lib` on Windows (as Administrator):
```powershell
Copy-Item "C:\Windows\System32\lxss\lib\libnvidia-gpucomp.so" `
"C:\Windows\System32\lxss\lib\libnvidia-gpucomp.so.595.54"
```
Then restart the nvidia-device-plugin pod. No AKS-EE restart required.
> **Note:** this workaround must be reapplied after every NVIDIA driver upgrade, as the version suffix changes with each driver version.
## Expected behavior
Either:
1. The WSL mount should expose `libnvidia-gpucomp.so` with a versioned filename consistent with its SONAME (as NVIDIA/Microsoft should fix), or
2. AKS-EE should handle the case where a versioned library path registered in ldconfig does not exist as a physical file, and fall back to the unversioned `.so`
Contributor guide
Research direction
No repository files or tests are named. Start by reproducing the nvidia-device-plugin daemonset failure and inspect the nvidia-container-runtime mount of /usr/lib/wsl/lib/libnvidia-gpucomp.so.595.54. Done means the plugin starts with NVIDIA drivers >= 595.x without requiring the manual PowerShell copy workaround, or the issue is confirmed to require an NVIDIA/Microsoft fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, linux, powershell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100