NVIDIA / NVIDIA/nvcf

docs(self-managed): document NVCA Helm resource-limit enforcement during upgrades

Open
#1,888 0 comments 0 reactions 1 assignee View on GitHub

@sbaum1994 is already working on this.

Since Sep 15, 2026.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Problem

Upgrading a self-managed deployment can begin rejecting existing customer Helm charts when NVCA CPU and memory resource-limit enforcement becomes active. The failure is reported by the NVCA validating webhook, for example:

admission webhook "validate-helm-charts.nvca.nvcf.nvidia.io" denied the request: container <name> has no resource limits

This is an upgrade compatibility requirement, but the self-managed upgrade documentation does not currently identify the version boundary, required chart changes, or the supported emergency opt-out.

The version history has two distinct boundaries that the documentation should explain:

  • NVCA added Helm function resource-limit enforcement in v2.47.0 and Helm task enforcement in v2.47.1. The NVCA feature flags themselves default to disabled.
  • The self-managed stack began enabling resource enforcement by default in v0.7.0 through #398. This stack policy, not the NVCA 3.8.0 binary alone, is what makes the constraint active during an upgrade. The first standalone compute-plane stack release containing the policy is v0.2.0.

This was observed during upgrade validation from a pre-enforcement self-managed stack using NVCA 3.0.4 to a newer stack using NVCA 3.8.0. User-supplied Helm function and task charts without limits were rejected deterministically, while a chart that declared limits succeeded.

Required documentation

Add an upgrade note and configuration guidance that covers all of the following:

  1. State the release boundaries above. Avoid describing this only as an NVCA 3.8.0 behavior change.

  2. Explain that every regular container and init container rendered by a Helm function or Helm task must declare both CPU and memory limits. Include a minimal example:

    resources:
      limits:
        cpu: "1"
        memory: 1Gi
    
  3. Explain that enforcement is controlled independently for Helm functions and Helm tasks by:

    • EnforceHelmFunctionResourceLimits
    • EnforceHelmTaskResourceLimits
  4. Document the persistent self-managed stack override for temporarily disabling either or both defaults:

    global:
      nvcaOperator:
        selfManaged:
          featureGateValues:
            - "-EnforceHelmFunctionResourceLimits"
            - "-EnforceHelmTaskResourceLimits"
    

    The leading - is required. The stack's feature-gate normalization explicitly preserves negative overrides. Clarify that disabling HelmResourceConstraints alone does not disable this CPU and memory validating-webhook check.

  5. Describe how to apply the values through the normal compute-plane upgrade workflow and verify the effective feature gates after the NVCA rollout.

  6. Recommend adding limits as the durable fix. Describe the negative overrides as a temporary compatibility escape hatch because disabling enforcement can reintroduce resource contention and eviction risk.

Update the current self-managed cluster configuration page and the applicable versioned upgrade or release-note page so customers encounter the requirement before upgrading.

BDD coverage

Add at least one customer-visible scenario under tests/bdd/ for this upgrade contract. The minimum scenario should install the compute plane with the default feature gates, deploy a Helm workload that omits CPU and memory limits, and assert that deployment fails with the NVCA resource-limit validation message.

Prefer covering the compatibility path as a second scenario: add the explicit negative Helm function/task feature-gate override, apply the compute-plane update, and verify that the same legacy Helm workload is no longer rejected by the resource-limit webhook. A compliant Helm chart should also remain covered by an end-to-end success path.

Keep the scenario in Gherkin and use the existing command/output DSL. Do not add a domain-specific step handler.

Acceptance criteria

  • The current and versioned self-managed documentation identifies NVCA v2.47.0 for Helm functions, NVCA v2.47.1 for Helm tasks, self-managed stack v0.7.0, and compute-plane stack v0.2.0 as the relevant boundaries.
  • The documentation shows the required CPU and memory limits for every regular and init container.
  • The documentation shows the exact negative feature-gate override and explains its temporary nature.
  • At least one tests/bdd scenario verifies the default enforcement behavior.
  • Existing Helm test fixtures that are intended to succeed with default enforcement declare compliant limits.

References

  • #397
  • #398

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.