nebari-dev / nebari-dev/data-science-pack
Auto-add nvidia.com/gpu toleration to GPU JupyterHub profiles
@aktech is already working on this.
Since Jun 15, 2026.
- Dominant language
- Python
- Stars
- 5
- Forks
- 7
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 13
Description
Background
nebari-dev/nebari-infrastructure-core#370 makes NIC automatically taint AWS GPU node groups with nvidia.com/gpu=true:NoSchedule, so ordinary pods stay off GPU nodes.
Nothing injects the matching toleration for us on EKS:
- The NVIDIA GPU Operator only tolerates this taint on its own operands (driver, device plugin), not on user pods.
- The Kubernetes
ExtendedResourceTolerationadmission controller would auto-tolerate pods that requestnvidia.com/gpu, but it is not enabled on EKS and cannot be enabled on the managed control plane. AWS's own GPU guide hand-writes the toleration on every example pod for this reason.
So once #370 lands, a JupyterHub GPU profile will stop scheduling new servers onto GPU nodes unless the profile carries the toleration.
Request
Auto-add a toleration for nvidia.com/gpu to any JupyterHub/KubeSpawner profile that targets GPUs, so users do not have to hand-write it. A GPU profile is one that requests nvidia.com/gpu in its resources, or is otherwise marked as a GPU profile (for example a GPU node selector). The injected toleration should be:
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
operator: Exists keeps it working regardless of the taint value, so it matches both NIC's value: "true" and other values.
Acceptance criteria
- A GPU profile launched against a cluster whose GPU nodes carry
nvidia.com/gpu=true:NoScheduleschedules onto the GPU node and starts. - Non-GPU profiles are unchanged and gain no toleration.
- Inside a running GPU server, the GPU is visible (
nvidia-smi, CUDA available to the framework).
Blocks
This blocks nebari-dev/nebari-infrastructure-core#370. The taint change should not roll out until GPU profiles tolerate the taint, otherwise GPU notebooks break.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.