NVIDIA / NVIDIA/nvcf

Complete PKI support for multi-cluster QUIC transport

Open
#19 4 comments 0 reactions 1 assignee View on GitHub

@mikeyrcamp is already working on this.

Since Jul 15, 2026.

enhancement roadmap
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Problem

Multi-cluster self-managed NVCF needs certificate-validated worker-to-control-
plane communication over QUIC. Load balancer and Gateway API termination do not
cover this path because the target deployments pass UDP through to the NVCF
request router.

The supported design uses cert-manager for service certificates, pluggable
issuers for managed and customer-owned PKI, and explicit public trust material
for compute-plane workers. Client identity remains PSAT/OIDC based. Mutual TLS
is not part of this epic.

Current status

The native hierarchy is 8 of 10 complete.

Completed:

  • #47 configures the default self-managed issuer contract.
  • #48 issues request-router TLS certificates.
  • #49 bootstraps transport trust through nvcf-cli.
  • #50 injects transport trust into NVCA-managed containers.
  • #51 supports external cert-manager and customer-managed issuers.
  • #54 replaces the BusyBox trust installer.
  • #838 validates advertised hostnames against certificate SANs.
  • #606 injects transport trust into Helm LLM utility pods.

Remaining native sub-issues:

  • #947 adds a pre-created request-router TLS Secret mode. It must preserve
    customer ownership of issuance and renewal, avoid rendering a cert-manager
    Certificate, and keep the public worker trust bundle separate from the
    server private key.
  • #52 records released-artifact end-to-end validation. It must prove a real
    LLM workload connects over certificate-validated reverse QUIC and that wrong
    or missing trust fails closed.

Required related work outside the native hierarchy:

  • PR #777 implements the server-identity reload portion of #599. It must
    merge before this epic closes. Client trust reload remains separate in #931.

Integration status

PR #951 merged and closed #950. The self-managed stack now has one
dependency-stage owner for the nvcf-pki Helm release. Preserve that ownership
model when implementing #947.

PR #777 is open and mergeable, but still requires review and completion of its
current checks. Its previous live-cluster validation exercised the earlier
filesystem watcher. Before merge, rerun the rotation scenarios against a real
kubelet using the current 30-second polling implementation. Confirm activation
within the polling window, repeated rotations recover from transient rejected
reads, and the expiry gauge advances for every valid rotation.

End-to-end gap

Issue #52 has no implementation or recorded validation evidence. Complete it
after #947 and PR #777 with:

  • An explicitly configured worker-reachable request-router endpoint.
  • A single request-router replica. Multi-replica chart-owned routing is
    deferred in #689.
  • A hostname used consistently for the worker address, TLS SNI, and certificate
    SAN.
  • Published charts and images rather than source-only render tests.
  • Positive worker registration and invocation evidence.
  • Wrong-root, wrong-hostname, missing-trust, and fingerprint-mismatch failures.
  • Server certificate rotation without a request-router restart.
  • Recovery after restoring valid trust material.

Deferred follow-up

These items do not block this epic, #52, or general availability:

  • #53: optional Restricted Pod Security hardening.
  • #584: gRPC RED metrics and tracing.
  • #931: client trust hot reload. Issue #599 remains the parent for both reload
    halves, but its server-identity portion in PR #777 is required now.
  • #946: strict rejection of unsupported Stargate server identities.
  • #689: chart-owned multi-replica routing.
  • #912: coordinated OpenBao network-policy hardening.

Completion criteria

  • Merge PR #951 and close #950.
  • Validate the current PR #777 polling path against a real kubelet, clear
    review and checks, and merge the server-identity reload change.
  • Complete and close #947.
  • Complete and close #52 with recorded released-artifact evidence.
  • Reconcile #599 after PR #777 merges. Keep the deferred client-trust work
    in #931 without treating the server-identity work as deferred.
  • Confirm every remaining open item is either a native completion gate or
    explicitly recorded as follow-up.
  • Close this epic only after the native hierarchy and this status agree.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.