NVIDIA / NVIDIA/nvcf

Self-hosted control-plane HA

Open
#985 0 comments 0 reactions 1 assignee View on GitHub

@shobham-nv is already working on this.

Since Aug 19, 2026.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

Enable in-cluster high availability for the NVCF self-hosted control plane so a single pod or node failure does not take down API, invocation, messaging or registration.

Delivers one Helmfile highAvailability: profile that turns on:

  • Tier-1 active–active (≥2 Ready replicas with anti-affinity, topology spread, PDB)
  • Tier-2 quorum HA (NATS 3, OpenBao 3, Cassandra 3; multi-AZ uses LOCAL_QUORUM)
  • Docs for AZ labels and dedicated node pools in both AZs
  • Basic failure validation (install shape, pod kill, drain)
  • Preserve single-node / local BDD behavior

Definition of Done

  • highAvailability: schema exists and maps through global.yaml.gotmpl to chart values
  • Production profile install on ≥3 nodes: Tier-1 ≥2 Ready; Tier-2 = 3 Ready on distinct nodes
  • Tier-1 pod kill / node drain does not cause sustained control-plane outage
  • single-node / local BDD path still works
  • Operator docs cover AZ labels, dedicated pools, and short RTO notes

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.