NVIDIA / NVIDIA/nvkind

extreme instabilities on well populated cluster

Open
#13 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
211
Forks
32
PR merge metrics
No merged PRs in 30d

Description

(wrongly felt) it works for us on single container, but it won't stand a reboot (see: https://github.com/kubernetes-sigs/kind/issues/2879#issuecomment-2472643645)

on Debian 12 (installing bookworm-backports for nvidia drivers) we have tons of Pods being
killed completely randomly... having a hard time to isolate the issue if not just mentioning the
OS we bootstrapped from (identical machine specs -> Hetzner's GEX44)

I initially assumed it did work on Ubuntu Jammy, but the reality was we were running on single container

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reboot failure on Debian 12 with bookworm-backports and the Hetzner GEX44 setup, comparing it with the single-container case. Inspect which Pods are killed and collect the relevant Kubernetes and NVIDIA driver logs. Done means isolating whether the OS or driver setup causes the random Pod terminations and documenting a reproducible cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
debian, kubernetes
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.