The GPU Operator Kind Demo Not Detecting NVIDIA Device
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 211
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
The ensure_gpu_present functions within the shell script gpu-operator-kind-demo.sh - GPU Operator Kind Demo does not appear to be working as expected.
The script halts when it's checking if a NVIDIA GPU device is present in the host:
./gpu-operator-kind-demo.sh
[demo] Installing packages: ca-certificates curl gnupg lsb-release pciutils tar
....
....
ca-certificates is already the newest version (20260601~24.04.1).
curl is already the newest version (8.5.0-2ubuntu10.11).
gnupg is already the newest version (2.4.4-2ubuntu17.4).
lsb-release is already the newest version (12.0-2).
pciutils is already the newest version (1:3.10.0-2build1).
tar is already the newest version (1.35+dfsg-3ubuntu0.3).
The following package was automatically installed and is no longer required:
linux-hwe-6.17-headers-6.17.0-40
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Synchronizing state of docker.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable docker
[demo][error] No NVIDIA GPU detected via lspci on this host.
This occurs even if there is a NVIDIA GPU device present in the host:
lspci | grep -i nvidia
04:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
It appears that the test condition that checks the presence of the NVIDIA device within the ensure_gpu_present function uses grep -q which causes lspci to fail with a 141 - EPIPE (Broken pipe) exit error even when grep matches the nvidia string.
Here is the scenario that illustrates the issue:
lspci | grep -qi nvidia; echo "lspci => ${PIPESTATUS[0]} grep => ${PIPESTATUS[1]}"
lspci => 141 grep => 0 # lspci returns EPIPE (Broken pipe) due to grep -q
Proposed solution (happy to submit a PR if it makes sense ):
If the grep program consumes entire output from the lspci program, the 141 - EPIPE (Broken pipe) error should not occur and the condition will only be false if grep does not match the "nvidia" string. Indicating the no NVIDIA GPU device was found.
So by removing the -q from the grep statement and instead piping the output to /dev/null, the function should work as expected.
# Happy Path
lspci | grep -i nvidia >/dev/null 2>&1; echo "lspci => ${PIPESTATUS[0]} grep => ${PIPESTATUS[1]}"
lspci => 0 grep => 0 # grep finds nvidia and doesn't cause Broken pipe
# Simulates "nvidia" not found using the incorrect string "nvid1a".
lspci | grep -i nvid1a >/dev/null 2>&1; echo "lspci => ${PIPESTATUS[0]} grep => ${PIPESTATUS[1]}"
lspci => 0 grep => 1 # grep did not find nvidia and doesn't cause Broken pipe
# Simulates "lspci" not available
l5pci | grep -i nvidia >/dev/null 2>&1; echo "lspci => ${PIPESTATUS[0]} grep => ${PIPESTATUS[1]}"
Command 'l5pci' not found, did you mean:
command 'lpci' from snap lpci (0.2.15)
command 'lspci' from deb pciutils (1:3.10.0-2)
See 'snap info <snapname>' for additional versions.
lspci => 127 grep => 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in examples/gpu-operator/gpu-operator-kind-demo.sh, at the ensure_gpu_present function and the test near line 175. Reproduce the check with lspci on a host containing an NVIDIA device, then verify the script distinguishes a matching device, no match, and a missing lspci command without the pipeline reporting EPIPE. Done means the demo no longer falsely reports that an available NVIDIA GPU is absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100