feat(stargate): hot reload client TLS trust bundles
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Why
#777 reloads mounted TLS server identities without a restart. Client trust
bundles still do not reload, so changing a CA requires restarting every workload
that dials with it. For Pylon that means a rolling restart of GPU worker pods,
which is the expensive half of a rotation.
This issue tracks the remaining half of #599.
Scope
The trust consumers that still pin their bundle at startup:
| Component | Role | Material |
|---|---|---|
| Pylon | Client dialing reverse-tunnel endpoints | CA / trust bundle |
| Stargate | Client dialing direct backends or relays | CA / trust bundle |
stargate-k8s-router |
Client dialing upstream Raw QUIC or WebTransport | CA / trust bundle |
Design constraints carried over from #777
The first attempt at this work was reverted from #777 because it grew a large
amount of machinery. Recording what to do differently:
- Use
quinn::Endpoint::set_default_client_configrather than replacing the
endpoint. The reverted version rebuilt wholeRelayEndpointsand
ClientEndpointsbehind anRwLockand closed the previous endpoint. That
rebinds UDP sockets, puts a lock and a lock-poisoned error path on the dial
hot path, and changes the source port.set_default_client_configapplies new
trust to subsequent dials with none of that. The code already calls it at
construction in a dozen places. - Use one per-generation
CancellationTokenfor the connection-closure
requirement, not awatchchannel with the sameselect!arm pasted into
each await point. The reverted version had that block copied three times into
a single function inwebtransport.rs. - Do not reintroduce a transactional multi-role commit. Server identity and
client trust are independent; a failure in one should not block the other.
stargate-tlsnow hasServerIdentityReloadTaskas the shape to follow. - Reuse
TlsMaterialChangeDetector. The notify watcher and its
reconciliation fallback are already shared and do not need a second copy. - Add
TlsMaterial::ClientTrust. Thematerial_typemetric label already
exists ontls_reloads_totalandtls_certificate_expiry_secondsfor exactly
this, so the metric contract does not change when trust reload lands.
Validate the initial bundle too, not just replacements
Pylon never validates its configured trust bundle. In secure reverse mode
std::fs::read is the only check, and the first thing that actually parses the
bytes is build_trusted_client_config inside connect_reverse_quic_endpoint,
which runs in run_reverse_tunnel_loop behind a 1s-to-30s backoff. A malformed,
empty, or anchor-free bundle therefore produces a Pylon that starts cleanly and
then fails every dial forever, with the reason buried in retry logging.
The other two consumers do not have this gap: stargate-k8s-router builds its
upstream client config in QuicRouterRuntime::bind and Stargate builds its in
ClientEndpoints::build, so a bad bundle fails startup. Pylon reverse mode is
the only path that defers the parse into a retry loop.
Whoever implements reload has to parse the bundle up front anyway to establish
the last-known-good baseline, so fixing this falls out of the same work. Parse
once during startup and fail there, the same way ServerIdentityReloader::load
already rejects a bad server identity in direct mode.
Fix the registration restart ordering before restarting a live session
InferenceServerRegistrationClient::start in
crates/pylon-lib/src/registration/client.rs stops the running session before
converting the supplied configuration:
pub fn start(&mut self, config: InferenceServerRegistrationConfig) -> Result<(), ClientError> {
self.stop();
let config = config.try_into()?;
...
A configuration that fails conversion therefore leaves running empty and
returns an error, so a rejected replacement takes the inference server out of
routing rather than being rejected. TryFrom rejects an empty seed list, a
direct-mode inference_server_url that is not quic://, and a reverse-mode URL
it cannot infer an http(s) base from.
This is latent on main and in #777, because the only caller is process startup
where running is already None and stop() is a no-op. It stops being latent
here: restarting the registration client is how the reverted implementation
swapped in a rotated trust bundle, so a bad bundle would have taken a healthy
worker offline instead of being rejected.
Convert first, stop only once the replacement is known good. It is a two-line
reorder. It was briefly carried in #777 and pulled back out, because there it
guarded a call path that does not exist.
Acceptance criteria
Carried from #599, limited to the trust half:
- Stargate direct and relay clients use an updated trust bundle for new
connections without a pod restart. - Pylon reverse-tunnel clients use an updated trust bundle for new connections
without a pod restart. stargate-k8s-routerreloads upstream trust for Raw QUIC and WebTransport.- An overlapping old-plus-new bundle supports planned CA rotation with no
insecure fallback. - Removing the old CA prevents new connections under the old root and closes
affected established connections before reconnecting under the replacement. - An invalid, empty, or anchor-free replacement is rejected and the
last-known-good bundle stays active. - Pylon fails startup when secure reverse mode is configured with a bundle that
is empty, malformed, or contains no usable trust anchor, rather than starting
and failing every dial behind a backoff. - Restarting the registration session with a configuration that fails
conversion leaves the existing session running, with a test covering it. - Wrong-root and wrong-SNI connections fail closed throughout.
- Tests cover projected-volume semantics, duplicate events, and trust
contraction closing established connections. - The Transport TLS Rotation runbook drops its "trust bundles do not reload"
scope section and documents emergency revocation without a restart.
Related
- #599 (parent)
- #777 (server identity half)
- #502
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing ServerIdentityReloader::load and TlsMaterialChangeDetector patterns, then trace ClientEndpoints::build, QuicRouterRuntime::bind, and Pylon's connect_reverse_quic_endpoint path. Also inspect crates/pylon-lib/src/registration/client.rs for the conversion-order fix. Done means all listed consumers reload validated trust bundles safely, affected connections reconnect correctly, and the registration and runbook requirements are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- infrastructure, networking, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100