NVIDIA / NVIDIA/nvcf

feat(stargate): hot reload client TLS trust bundles

Open
#931 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Why

#777 reloads mounted TLS server identities without a restart. Client trust
bundles still do not reload, so changing a CA requires restarting every workload
that dials with it. For Pylon that means a rolling restart of GPU worker pods,
which is the expensive half of a rotation.

This issue tracks the remaining half of #599.

Scope

The trust consumers that still pin their bundle at startup:

Component Role Material
Pylon Client dialing reverse-tunnel endpoints CA / trust bundle
Stargate Client dialing direct backends or relays CA / trust bundle
stargate-k8s-router Client dialing upstream Raw QUIC or WebTransport CA / trust bundle

Design constraints carried over from #777

The first attempt at this work was reverted from #777 because it grew a large
amount of machinery. Recording what to do differently:

  • Use quinn::Endpoint::set_default_client_config rather than replacing the
    endpoint.
    The reverted version rebuilt whole RelayEndpoints and
    ClientEndpoints behind an RwLock and closed the previous endpoint. That
    rebinds UDP sockets, puts a lock and a lock-poisoned error path on the dial
    hot path, and changes the source port. set_default_client_config applies new
    trust to subsequent dials with none of that. The code already calls it at
    construction in a dozen places.
  • Use one per-generation CancellationToken for the connection-closure
    requirement
    , not a watch channel with the same select! arm pasted into
    each await point. The reverted version had that block copied three times into
    a single function in webtransport.rs.
  • Do not reintroduce a transactional multi-role commit. Server identity and
    client trust are independent; a failure in one should not block the other.
    stargate-tls now has ServerIdentityReloadTask as the shape to follow.
  • Reuse TlsMaterialChangeDetector. The notify watcher and its
    reconciliation fallback are already shared and do not need a second copy.
  • Add TlsMaterial::ClientTrust. The material_type metric label already
    exists on tls_reloads_total and tls_certificate_expiry_seconds for exactly
    this, so the metric contract does not change when trust reload lands.

Validate the initial bundle too, not just replacements

Pylon never validates its configured trust bundle. In secure reverse mode
std::fs::read is the only check, and the first thing that actually parses the
bytes is build_trusted_client_config inside connect_reverse_quic_endpoint,
which runs in run_reverse_tunnel_loop behind a 1s-to-30s backoff. A malformed,
empty, or anchor-free bundle therefore produces a Pylon that starts cleanly and
then fails every dial forever, with the reason buried in retry logging.

The other two consumers do not have this gap: stargate-k8s-router builds its
upstream client config in QuicRouterRuntime::bind and Stargate builds its in
ClientEndpoints::build, so a bad bundle fails startup. Pylon reverse mode is
the only path that defers the parse into a retry loop.

Whoever implements reload has to parse the bundle up front anyway to establish
the last-known-good baseline, so fixing this falls out of the same work. Parse
once during startup and fail there, the same way ServerIdentityReloader::load
already rejects a bad server identity in direct mode.

Fix the registration restart ordering before restarting a live session

InferenceServerRegistrationClient::start in
crates/pylon-lib/src/registration/client.rs stops the running session before
converting the supplied configuration:

pub fn start(&mut self, config: InferenceServerRegistrationConfig) -> Result<(), ClientError> {
    self.stop();
    let config = config.try_into()?;
    ...

A configuration that fails conversion therefore leaves running empty and
returns an error, so a rejected replacement takes the inference server out of
routing rather than being rejected. TryFrom rejects an empty seed list, a
direct-mode inference_server_url that is not quic://, and a reverse-mode URL
it cannot infer an http(s) base from.

This is latent on main and in #777, because the only caller is process startup
where running is already None and stop() is a no-op. It stops being latent
here: restarting the registration client is how the reverted implementation
swapped in a rotated trust bundle, so a bad bundle would have taken a healthy
worker offline instead of being rejected.

Convert first, stop only once the replacement is known good. It is a two-line
reorder. It was briefly carried in #777 and pulled back out, because there it
guarded a call path that does not exist.

Acceptance criteria

Carried from #599, limited to the trust half:

  1. Stargate direct and relay clients use an updated trust bundle for new
    connections without a pod restart.
  2. Pylon reverse-tunnel clients use an updated trust bundle for new connections
    without a pod restart.
  3. stargate-k8s-router reloads upstream trust for Raw QUIC and WebTransport.
  4. An overlapping old-plus-new bundle supports planned CA rotation with no
    insecure fallback.
  5. Removing the old CA prevents new connections under the old root and closes
    affected established connections before reconnecting under the replacement.
  6. An invalid, empty, or anchor-free replacement is rejected and the
    last-known-good bundle stays active.
  7. Pylon fails startup when secure reverse mode is configured with a bundle that
    is empty, malformed, or contains no usable trust anchor, rather than starting
    and failing every dial behind a backoff.
  8. Restarting the registration session with a configuration that fails
    conversion leaves the existing session running, with a test covering it.
  9. Wrong-root and wrong-SNI connections fail closed throughout.
  10. Tests cover projected-volume semantics, duplicate events, and trust
    contraction closing established connections.
  11. The Transport TLS Rotation runbook drops its "trust bundles do not reload"
    scope section and documents emergency revocation without a restart.

Related

  • #599 (parent)
  • #777 (server identity half)
  • #502

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing ServerIdentityReloader::load and TlsMaterialChangeDetector patterns, then trace ClientEndpoints::build, QuicRouterRuntime::bind, and Pylon's connect_reverse_quic_endpoint path. Also inspect crates/pylon-lib/src/registration/client.rs for the conversion-order fix. Done means all listed consumers reload validated trust bundles safely, affected connections reconnect correctly, and the registration and runbook requirements are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
infrastructure, networking, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.