docs(self-managed): define a CSP-agnostic multi-region control-plane reference architecture

Description

Track deliverable 2 of #41 and the architecture decisions required by #1375.

Define a production reference architecture for a full multi-region self-managed NVCF control plane. The design must cover the platform services and their state, not only compute clusters in different regions. NATS and Cassandra are central to the design, and the same analysis must include global routing, secret and certificate distribution, service discovery, upgrades, observability, and failure recovery.

The target is operational parity with the current NVIDIA-managed five-region NVCF deployment. This reference architecture should provide the provider-neutral deployment model that can eventually replace the current managed bring-up style.

ELI5 target behavior

Treat each region like a self-sufficient local shop. It should keep serving nearby customers when the network to another region is unavailable. A small global directory can tell customers and operators which shops exist and coordinate shared changes, but one broken shop or inter-region link must not close every shop.

The architecture must say which information each shop owns, what is copied between shops, how quickly it is copied, and what happens when two shops cannot talk to each other.

Initial working hypothesis from the spike

The following is a starting hypothesis, not a final decision:

• Use repeatable regional cells for latency-sensitive invocation, routing, scheduling, and regional state.
• Keep the global management layer as small as practical. Regional serving paths should continue during an inter-region partition when their required local dependencies remain healthy.
• Run a highly available NATS cluster within each region. Evaluate NATS supercluster gateways for inter-region routing, while keeping JetStream replicas local through explicit placement. Define mirrors, sources, or application-level replication only where cross-region message movement is required.
• Do not stretch one NATS Raft group across a wide-area network without measured evidence and an explicit failure contract.
• Decide Cassandra topology from data ownership, consistency, latency, and recovery requirements. Candidate designs include one multi-datacenter cluster, independent regional clusters with application-level replication, and a hybrid split by data class.
• Give every logical region one provider-neutral identity. Store cloud provider and provider region as separate optional metadata.
• Replace static peer lists with a supported region registry or discovery contract as the number of regions grows.

Required architecture decisions

Global and regional boundaries

• Define the global management plane, each regional control-plane cell, and the compute-plane boundary.
• Classify every service as global, regional, cell-local, or external.
• Define whether a logical region can contain more than one control-plane cell.
• Define region registration, discovery, health, drain, removal, and replacement.
• Identify every synchronous request that crosses a region and justify why it cannot be local or asynchronous.

Data ownership and consistency

• Classify function metadata, deployment state, cluster state, credentials, secrets, routing state, queue state, invocation metadata, usage data, and audit data.
• Assign a system of record and write owner for every data class.
• Define consistency, conflict resolution, replication, retention, recovery point objective, and recovery time objective per data class.
• Define behavior during regional loss, inter-region partition, high latency, partial replication, and recovery.
• Prevent split-brain writes where strong global ordering is required.

NATS and JetStream

• Select the supported cluster, gateway, domain, account, and placement topology.
• Define server counts, availability-zone placement, quorum boundaries, and storage requirements in each region.
• Classify streams and consumers as regional or global and define their ownership.
• Define which messages may cross regions and whether gateways, stream sources or mirrors, or application logic provide that movement.
• Define behavior for gateway loss, regional quorum loss, slow consumers, replay, duplicate delivery, and recovery.
• Define how placement tags are derived from the provider-neutral region contract in #1375.
• Provide preflight checks that prove every required stream can be placed before a region is admitted.

Cassandra

• Compare at least these options:
  • One Cassandra cluster with a datacenter per NVCF region and NetworkTopologyStrategy replication.
  • Independent Cassandra clusters per regional cell with explicit replication or event propagation.
  • A hybrid model that separates global metadata from regional operational state.
• For each option, document read and write consistency, local and remote quorum behavior, latency, failure isolation, repair, backup, restore, schema migration, scaling, and region removal.
• Define replication factors and consistency levels from the required failure contract rather than hard-coded region names.
• Define safe bootstrap, expansion, repair, decommission, and replacement procedures.
• Validate behavior under realistic inter-region latency and partitions before selecting a topology.

Routing and external access

• Define global and regional ingress, DNS or global load balancing, locality, health signals, failover, failback, and connection draining.
• Cover HTTP, gRPC, streaming, callbacks, LLM routing, API access, and cluster registration paths.
• Define whether clients select a region or use a global endpoint and how routing state converges.
• Align with the scalable Pylon discovery work in #1328.

Secrets, identity, and artifacts

• Define OpenBao or external vault topology, unseal and recovery, certificate authorities, token validation, key rotation, and regional secret availability.
• Define artifact and image availability when a region is isolated.
• Avoid a global secret or identity dependency on the regional invocation hot path unless its failure behavior is explicit.

Delivery and operations

• Define how Argo Rollouts operates inside each region and how release orchestration coordinates promotion, pause, abort, and rollback across regions.
• Define supported control-plane version skew and database or message-schema compatibility during staged regional upgrades.
• Define observability aggregation without making regional service health depend on a remote telemetry backend.
• Provide per-region and global readiness, capacity, saturation, replication-lag, and quorum signals.
• Define backup, disaster recovery, game-day, and operator runbook requirements.

CSP-agnostic infrastructure contract

• Define required capabilities for networking, load balancing, DNS, block storage, object storage, secrets, identity, and Kubernetes without assuming one CSP's resource names.
• Keep provider adapters behind stable interfaces and document the capability checks needed during bring-up.
• Provide at least one generic or on-premises reference mapping and cloud mappings without changing NVCF's logical region model.

Alternatives to compare

• One control-plane stack stretched across all regions.
• A complete independent NVCF stack in every region with no global coordination.
• Regional cells with a thin global management layer.
• A hybrid where selected global state is replicated while invocation and operational state remains regional.

The decision record must explain the tradeoffs in availability, consistency, latency, cost, operational complexity, and migration risk. It should not select an option solely because it resembles one CSP's managed service layout.

Deliverables

• Architecture document with deployment, request-flow, state-flow, rollout, and failure-recovery diagrams.
• Architecture decision records for the regional cell boundary, NATS, Cassandra, routing, secrets, and region discovery.
• Service placement inventory showing global, regional, cell-local, and external components.
• Data ownership and consistency table for every persistent data class.
• Failure-mode matrix covering node, availability zone, region, inter-region link, quorum, routing, identity, and observability failures.
• Reference provider-neutral values or inventory for two-region and five-region deployments.
• Network, storage, quorum, latency, capacity, port, certificate, and identity prerequisites.
• Argo Rollouts and version-skew strategy for safe region-by-region upgrades.
• Proof-of-concept results for NATS and Cassandra under representative latency, partition, failover, recovery, and region expansion scenarios.
• Migration plan from the current five-region managed bring-up, including coexistence, state movement, traffic cutover, rollback, and retirement.
• Public operator guidance for supported topologies and explicit unsupported configurations.

Definition of done

• The chosen architecture supports at least five regions without a full mesh or other unbounded per-region configuration requirement unless that tradeoff is explicitly accepted.
• Regional invocation continues for a documented set of operations during an inter-region partition.
• The NATS design defines regional quorum, placement, cross-region message movement, and recovery behavior.
• The Cassandra design defines topology, replication, consistency, repair, schema migration, backup, restore, and region lifecycle.
• Global and regional state ownership is explicit, with documented consistency and conflict behavior.
• Argo Rollouts behavior and cross-region release orchestration are documented and validated against the configuration contract in #1375.
• The architecture is CSP agnostic and has a capability-based infrastructure contract.
• The reference topology is compared with the current managed five-region deployment, and every material gap is resolved, accepted, or linked to an owned follow-up issue.
• A migration and rollback path explains how this bring-up model can replace the current managed deployment style.
• Failure tests or proof-of-concept evidence validate the highest-risk NATS, Cassandra, routing, and rollout assumptions.
• The final public documentation is linked from the self-managed installation and operations guides.

Related work

• Parent epic: #41
• Configuration implementation: #1375
• Public roadmap: #27
• Short-term NATS placement fix: #1365
• Single-region control-plane resiliency: #1063
• Customer-managed infrastructure dependencies: #44
• Pylon multi-region discovery design: #1328
• NATS supercluster topology: https://docs.nats.io/learn/topologies/super-clusters
• NATS JetStream stream configuration: https://docs.nats.io/reference/jetstream/api/stream/create

Start with the configuration contract in #1375, the Pylon discovery work in #1328, and the linked NATS topology references; compare them with the current self-managed and managed deployment documentation. Produce the architecture, decision records, service and data ownership inventories, failure matrix, proof-of-concept evidence, and migration guidance described in the deliverables and definition of done.