nebari-dev / nebari-dev/llm-serving-pack
Add nodeSelector, tolerations and imagePullSecrets for the EPP under spec.advanced.inferencePool
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3
- Forks
- 3
- Avg merge
- 16h 31m
- Merged PRs (30d)
- 11
Description
Problem
InferencePoolAdvancedSpec (operator/api/v1alpha1/llmmodel_types.go) exposes only schedulerConfig. nodeSelector, affinity and tolerations exist only under spec.advanced.vllm and apply to the model pod. There is no way to place the EPP Deployment or give it imagePullSecrets.
Customer report: a two-node cluster (one GPU node, one CPU node) where the EPP image became unpullable (https://github.com/Concepts-NREC/Nebari-GitOps, see the llm-serving-pack default-image issue). One EPP replica ran on the GPU node with the image already present; the other sat in ImagePullBackOff on the CPU node. They could not pin the EPP to the node that had the image, and hand-patching the EPP Deployment was reverted by the operator within ~90s.
Proposal
Add to InferencePoolAdvancedSpec:
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
Tolerations []corev1.Toleration `json:"tolerations,omitempty"`
Affinity *corev1.Affinity `json:"affinity,omitempty"`
ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
Replicas *int32 `json:"replicas,omitempty"`
and thread them through buildEPPDeployment in reconcilers/inferencepool.go. Replicas is included because a single-node or two-node cluster has no reason to run two EPPs; today the count is hardcoded.
Also consider an operator-level default (defaults.epp.imagePullSecrets, defaults.epp.nodeSelector) alongside defaults.epp.image, since a private mirror is a cluster-wide decision rather than a per-model one.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with InferencePoolAdvancedSpec in operator/api/v1alpha1/llmmodel_types.go and the buildEPPDeployment function in reconcilers/inferencepool.go. Trace how schedulerConfig reaches the EPP Deployment, then thread the proposed scheduling, pull-secret, and replica fields through the same path. Done means the EPP Deployment reflects these spec settings and operator reconciliation preserves them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100