NVIDIA / NVIDIA/gpu-operator

[Feature]: Support affinity / nodeSelector for operand DaemonSets in ClusterPolicy spec.daemonsets

Open
#2,626 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.9k
Forks
552
Avg merge
2d 4h
Merged PRs (30d)
90

Description

Feature description

Add affinity and nodeSelector to the common DaemonSet config in ClusterPolicy
(spec.daemonsets), surfaced in the Helm chart as daemonsets.affinity /
daemonsets.nodeSelector (default empty). When set, apply them to all operand
DaemonSets (driver, container-toolkit, device-plugin, dcgm-exporter, gfd,
mig-manager, …), on top of the operator's existing NFD-based node selection.

Motivation

spec.daemonsets already exposes tolerations but has no affinity/nodeSelector.
That's an asymmetry: tolerations only permit scheduling onto tainted nodes — they
can't restrict where operands land. On a cloud cluster (EKS/GKE/AKS) with several
autoscaled node pools where only one has GPUs, I want to declaratively pin operands to
that pool via a label it already carries (e.g. karpenter.sh/nodepool=gpu).

The documented alternatives don't cover this well:

  • nvidia.com/gpu.deploy.<operand>=disabled labels / node taints (how #547 was
    closed) are subtractive: you must label or taint every node you want to exclude.
    In autoscaled clusters (Karpenter, managed nodegroups) where nodes churn constantly,
    that doesn't scale and is race-prone.
  • NFD gating keeps operands off non-GPU nodes, but can't target a specific GPU
    pool and doesn't apply when NFD is disabled.

A first-class affinity/nodeSelector is declarative, applies automatically to new
nodes, and is consistent with the operator's own NvidiaDriver CRD, which already
exposes spec.affinity and spec.nodeSelector.

Prior art

#547 requested essentially this and was closed pointing at the subtractive workarounds
above; a community MR to add nodeSelector to the common daemonset config was referenced
there but never landed.

Proposed API

# ClusterPolicy
spec:
  daemonsets:
    nodeSelector:            # proposed
      karpenter.sh/nodepool: gpu
    affinity: {}             # proposed
# Helm values.yaml
daemonsets:
  affinity: {}       # default: unchanged behavior
  nodeSelector: {}   # default: unchanged behavior

User-supplied values should be merged (AND) with the operator's NFD-based selection so
GPU detection still applies.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ClusterPolicy spec.daemonsets API and the Helm daemonsets values, then trace how the common DaemonSet configuration reaches each listed operand. Verify that empty defaults preserve current behavior and that configured affinity and nodeSelector apply to all operand DaemonSets alongside existing NFD selection. Done means the API and chart expose both fields and the behavior is covered across the operands.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.