NVIDIA / NVIDIA/NeMo-Retriever

[BUG]: Helm chart NIMCache templates missing nodeSelector and tolerations fields

Open Beginner friendly
#1,636 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Version: 26.1.2
Installation method: Kubernetes/Helm (NIM Operator mode)
Describe the bug:
The NIMCache templates in the nv-ingest Helm chart (version 26.1.2) do not render nodeSelector or tolerations fields, even when these values are provided in the Helm values file. This causes NIMCache pods to be unschedulable on clusters where GPU nodes have taints or require specific node selection.
Looking at the template templates/llama-3.2-nv-embedqa-1b-v2.yaml, the NIMService section correctly renders nodeSelector and tolerations:

kind: NIMService
spec:
  nodeSelector:
{{ toYaml .Values.nimOperator.embedqa.nodeSelector | nindent 4 }}
  tolerations:
{{ toYaml .Values.nimOperator.embedqa.tolerations | nindent 4 }}

However, the NIMCache section in the same template only renders source and storage, omitting nodeSelector and tolerations entirely:

kind: NIMCache
spec:
  source:
    ngc:
      modelPuller: "..."
      pullSecret: "..."
      authSecret: ...
  storage:
    pvc:
      ...

Expected behavior:
The NIMCache templates should include nodeSelector and tolerations fields, similar to NIMService:

kind: NIMCache
spec:
  source:
    ...
  storage:
    ...
  nodeSelector:
{{ toYaml .Values.nimOperator.embedqa.nodeSelector | nindent 4 }}
  tolerations:
{{ toYaml .Values.nimOperator.embedqa.tolerations | nindent 4 }}

Workaround:
Currently requires manually patching each NIMCache resource after Helm deployment:

kubectl patch nimcache <name> -n nim --type=merge -p '{
  "spec": {
    "nodeSelector": {"cloud.google.com/gke-nodepool": "gpu-pool"},
    "tolerations": [{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}]
  }
}'

Affected templates:

  • templates/llama-3.2-nv-embedqa-1b-v2.yaml
  • templates/nemoretriever-graphic-elements-v1.yaml
  • templates/nemoretriever-ocr-v1.yaml
  • templates/nemoretriever-page-elements-v3.yaml
  • templates/nemoretriever-table-structure-v1.yaml

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the five affected templates: templates/llama-3.2-nv-embedqa-1b-v2.yaml, templates/nemoretriever-graphic-elements-v1.yaml, templates/nemoretriever-ocr-v1.yaml, templates/nemoretriever-page-elements-v3.yaml, and templates/nemoretriever-table-structure-v1.yaml. Compare each NIMCache section with its NIMService section and verify the rendered Helm output. Done means every affected NIMCache includes the configured nodeSelector and tolerations fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.