epic(llm-router): secure and reliable Pylon registration across clusters

Description

Make Pylon discovery and registration secure and reliable across self-managed and managed multi-cluster NVCF deployments.

The current failure chain contains independent client, ingress, discovery, and configuration defects. They need separate implementation and validation so one failure does not hide the next:

• HTTPS Stargate gRPC dial endpoints are constructed without tonic client TLS support.
• Pylon gRPC needs to reuse the existing system-plus-custom trust bundle contract.
• Self-managed worker registration on port 50071 has no supported TLS/h2 termination and certificate contract by default.
• Deployment-backed SRV discovery can produce an unroutable dashed-IP authority alongside the real pod identity.
• Scheme-less cross-region watch addresses normalize to plaintext HTTP even when the listener requires TLS.

Keep TLS dial identity separate from routing authority: certificate verification and SNI use the external dial hostname, while HTTP/2 :authority continues to carry the concrete Stargate identity selected by stargate-k8s-router.

Definition of Done

• HTTPS Pylon watch and registration channels use verified TLS and preserve plaintext behavior only for explicit HTTP endpoints.
• Pylon supports public/system roots and the existing --tls-cert-path / STARGATE_TLS_CERT_PATH merged trust bundle without an insecure fallback.
• LLM/PKI-enabled self-managed profiles expose worker-reachable TLS/h2 on port 50071 with a certificate covering the dial hostname and h2c forwarding to the backend router.
• The NVCF API bootstrap address and all WatchStargates dial advertisements use explicit secure URIs in secure profiles.
• StatefulSet- and Deployment-backed discovery are both supported first-class and emit only
  router-addressable Stargate identities; requiring one workload kind is not an acceptable fix.
• Multi-cluster tests independently cover TLS, authority routing, Deployment discovery, cross-region watch, registration, authenticated reverse QUIC, and invocation.
• Negative tests prove wrong-root, wrong-hostname, missing-trust, plaintext-to-TLS, and invalid authority cases fail closed.
• Documentation distinguishes gRPC TLS on port 50071 from reverse QUIC TLS on port 50072.

Workstreams

• #1293 — Enable verified TLS for HTTPS Pylon Watch and Register channels.
• #1281 — Reuse the existing merged system-plus-custom trust bundle for Stargate gRPC.
• #1294 — Provide supported TLS/h2 termination and certificate wiring for port 50071.
• #1295 — Support canonical router-addressable identities for both StatefulSets and Deployments.
• #1296 — Require explicit schemes for secure cross-region watch URLs.

These are also attached as native GitHub sub-issues. Existing issue #689 remains related to split-cluster request-router exposure rather than duplicating that network-plumbing scope here.

Resources

• Related: #19
• Related: #689
• Related but separate NVCF API authorization listener work: #1207

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow the project code of conduct and contributing guidelines.

Start by reading sub-issues #1293, #1281, #1294, #1295, and #1296, then map their implementation and test entry points in the repository. Treat the workstreams independently and validate each against the Definition of Done, including secure URI handling, certificate and authority behavior, discovery, registration, invocation, and negative multi-cluster tests.