NVIDIA / NVIDIA/NeMo-Retriever
[BUG]: Helm chart NIMCache templates missing nodeSelector and tolerations fields
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Version: 26.1.2
Installation method: Kubernetes/Helm (NIM Operator mode)
Describe the bug:
The NIMCache templates in the nv-ingest Helm chart (version 26.1.2) do not render nodeSelector or tolerations fields, even when these values are provided in the Helm values file. This causes NIMCache pods to be unschedulable on clusters where GPU nodes have taints or require specific node selection.
Looking at the template templates/llama-3.2-nv-embedqa-1b-v2.yaml, the NIMService section correctly renders nodeSelector and tolerations:
kind: NIMService
spec:
nodeSelector:
{{ toYaml .Values.nimOperator.embedqa.nodeSelector | nindent 4 }}
tolerations:
{{ toYaml .Values.nimOperator.embedqa.tolerations | nindent 4 }}
However, the NIMCache section in the same template only renders source and storage, omitting nodeSelector and tolerations entirely:
kind: NIMCache
spec:
source:
ngc:
modelPuller: "..."
pullSecret: "..."
authSecret: ...
storage:
pvc:
...
Expected behavior:
The NIMCache templates should include nodeSelector and tolerations fields, similar to NIMService:
kind: NIMCache
spec:
source:
...
storage:
...
nodeSelector:
{{ toYaml .Values.nimOperator.embedqa.nodeSelector | nindent 4 }}
tolerations:
{{ toYaml .Values.nimOperator.embedqa.tolerations | nindent 4 }}
Workaround:
Currently requires manually patching each NIMCache resource after Helm deployment:
kubectl patch nimcache <name> -n nim --type=merge -p '{
"spec": {
"nodeSelector": {"cloud.google.com/gke-nodepool": "gpu-pool"},
"tolerations": [{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}]
}
}'
Affected templates:
- templates/llama-3.2-nv-embedqa-1b-v2.yaml
- templates/nemoretriever-graphic-elements-v1.yaml
- templates/nemoretriever-ocr-v1.yaml
- templates/nemoretriever-page-elements-v3.yaml
- templates/nemoretriever-table-structure-v1.yaml
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the five affected templates: templates/llama-3.2-nv-embedqa-1b-v2.yaml, templates/nemoretriever-graphic-elements-v1.yaml, templates/nemoretriever-ocr-v1.yaml, templates/nemoretriever-page-elements-v3.yaml, and templates/nemoretriever-table-structure-v1.yaml. Compare each NIMCache section with its NIMService section and verify the rendered Helm output. Done means every affected NIMCache includes the configured nodeSelector and tolerations fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100