NVIDIA / NVIDIA/OpenShell

task: assess Kubernetes GPU support preflight validation

Open
#1,807 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:stale
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Description

Assess how the Kubernetes compute driver should determine whether a cluster exposes GPU resources before accepting GPU sandbox requests.

This follows from the --gpu-count work and should not change behavior in the current small CLI PR. The goal is to decide what signal the driver should use for fail-fast feedback when a cluster has no GPU support, without incorrectly rejecting requests that Kubernetes could schedule later.

Context

Related roadmap: #1444

The current Kubernetes driver preflight checks node status.allocatable["nvidia.com/gpu"] and treats any non-zero value as GPU support. During review of the GPU count CLI work, we identified that this check is really about whether the cluster exposes GPU resources, not whether a particular sandbox can schedule immediately.

Important distinctions:

  • capacity is a more stable signal that a node advertises the GPU extended resource.
  • allocatable is closer to whether pods can request that resource.
  • Neither field alone answers whether a specific pod can schedule right now.
  • Current resource pressure should generally be left to the Kubernetes scheduler, because a sandbox may become schedulable after other pods release GPUs.
  • Autoscaled GPU node pools may complicate any check based only on currently present nodes.

Definition of Done

  • Decide whether the Kubernetes preflight should use node capacity, node allocatable, both, or another signal.
  • Decide whether GPU support detection should remain boolean or account for requested gpu_count.
  • Decide how zero-scaled or autoscaled GPU node pools should be handled.
  • Document the intended semantics in the Kubernetes driver README or architecture docs.
  • Create implementation follow-up issues if behavior should change.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the current Kubernetes driver preflight check that inspects node status.allocatable and the related --gpu-count work. Compare capacity, allocatable, requested gpu_count, and autoscaled or zero-scaled pools against the stated semantics. Document the decision in the Kubernetes driver README or architecture docs, and record implementation follow-up issues if needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
infrastructure
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.