NVIDIA / NVIDIA/nvkind

Unable to find GPU in pod where GPU has been successfully allocated, and nvidia-smi displays the message "Device not found" after execution.

Open
#63 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
211
Forks
32
PR merge metrics
No merged PRs in 30d

Description

I used the nvkind to build a k8s cluster with two worker nodes as the example showed in a linux server.
a work node with 2 gpus and the other with 3 nodes.
Then I ininstalled a gpu-operator and everything looked fine.
After that, I tried to apply a pod via kubectl apply command.
This pod needed a gpu resource and the status of this pod was running.
Finally, I tried to log this pod to see if everything was ok and find that the nvidia-smi displayed the message "Device not found" after execution.

  1. The k8s cluster info:
    nvkind cluster print-gpus
    [
    {
    "node": "explicit-gpus-worker",
    "gpus": [
    {
    "Index": "0",
    "Name": "NVIDIA H20",
    "UUID": "GPU-3c10a6fd-a8c7-4e9a-0788-32b503a11514"
    },
    {
    "Index": "1",
    "Name": "NVIDIA H20",
    "UUID": "GPU-4fca9ef6-4ebf-f2ca-3d57-51b2dd23f146"
    }
    ]
    },
    {
    "node": "explicit-gpus-worker2",
    "gpus": [
    {
    "Index": "0",
    "Name": "NVIDIA H20",
    "UUID": "GPU-d7adcefb-078a-69d7-d3d1-3b853ce93225"
    },
    {
    "Index": "1",
    "Name": "NVIDIA H20",
    "UUID": "GPU-734f87a0-26b1-e9c7-501a-be3d8b85fb8e"
    },
    {
    "Index": "2",
    "Name": "NVIDIA H20",
    "UUID": "GPU-4917c52b-1af2-a05f-100f-d348f9cd7cf5"
    }
    ]
    }
    ]

  2. kubectl get pods
    NAME READY STATUS RESTARTS AGE
    gpu-pod 1/1 Running 0 45s

  3. kubectl logs gpu-pod
    No devices were found

4.gpu-pod yaml file
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
labels:
app: gpu-workload
spec:
nodeSelector:
accelerator: nvidia-tesla-k160 # Select nodes with GPU (adjust label as needed)
containers:
- name: gpu-container
image: nvidia/cuda:11.8.0-base-ubuntu22.04
command: ["bash", "-c"]
args: ["nvidia-smi; trap 'exit 0' TERM; sleep infinity & wait"]
resources:
requests:
nvidia.com/gpu: "1" # Request 1 GPU
limits:
nvidia.com/gpu: "1" # Limit to 1 GPU
restartPolicy: Never

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report with the nvkind cluster, the gpu-operator, the supplied gpu-pod YAML, and the nvkind cluster print-gpus, kubectl get pods, and kubectl logs gpu-pod commands. Inspect the node selection, GPU resource allocation, and container output to identify why nvidia-smi reports no devices; done means the cause is confirmed and GPU visibility works in the pod or the limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.