nebari-dev / nebari-dev/llm-serving-pack

Add nodeSelector, tolerations and imagePullSecrets for the EPP under spec.advanced.inferencePool

Open
#176 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs: discussion 💬
Dominant language
Go
Stars
3
Forks
3
Avg merge
16h 31m
Merged PRs (30d)
11

Description

Problem

InferencePoolAdvancedSpec (operator/api/v1alpha1/llmmodel_types.go) exposes only schedulerConfig. nodeSelector, affinity and tolerations exist only under spec.advanced.vllm and apply to the model pod. There is no way to place the EPP Deployment or give it imagePullSecrets.

Customer report: a two-node cluster (one GPU node, one CPU node) where the EPP image became unpullable (https://github.com/Concepts-NREC/Nebari-GitOps, see the llm-serving-pack default-image issue). One EPP replica ran on the GPU node with the image already present; the other sat in ImagePullBackOff on the CPU node. They could not pin the EPP to the node that had the image, and hand-patching the EPP Deployment was reverted by the operator within ~90s.

Proposal

Add to InferencePoolAdvancedSpec:

NodeSelector     map[string]string             `json:"nodeSelector,omitempty"`
Tolerations      []corev1.Toleration           `json:"tolerations,omitempty"`
Affinity         *corev1.Affinity              `json:"affinity,omitempty"`
ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
Replicas         *int32                        `json:"replicas,omitempty"`

and thread them through buildEPPDeployment in reconcilers/inferencepool.go. Replicas is included because a single-node or two-node cluster has no reason to run two EPPs; today the count is hardcoded.

Also consider an operator-level default (defaults.epp.imagePullSecrets, defaults.epp.nodeSelector) alongside defaults.epp.image, since a private mirror is a cluster-wide decision rather than a per-model one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with InferencePoolAdvancedSpec in operator/api/v1alpha1/llmmodel_types.go and the buildEPPDeployment function in reconcilers/inferencepool.go. Trace how schedulerConfig reaches the EPP Deployment, then thread the proposed scheduling, pull-secret, and replica fields through the same path. Done means the EPP Deployment reflects these spec settings and operator reconciliation preserves them.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.