feat(self-managed): add provider-neutral region configuration across the stack

Description

Track the configuration and bring-up work needed for deliverable 1 of #41.

Self-managed NVCF currently assumes one logical control-plane region and does not use one consistent region contract across the stack. #1365 fixes the immediate NATS placement failure by adding a compatibility tag, but that is only a short-term bridge.

The long-term goal is full multi-region self-managed NVCF with operational parity against the current NVIDIA-managed five-region NVCF deployment. This deployment model is intended to replace the current managed bring-up style over time. The implementation must therefore identify and close parity gaps instead of creating a separate, reduced multi-region mode.

The configuration must be cloud service provider (CSP) agnostic. A logical NVCF region must not be named or modeled as an AWS region, even when a deployment happens to run on AWS.

ELI5 target behavior

An operator should be able to give each regional control plane a stable name, such as region-a or region-b, and install the same stack in each region. Every NVCF component should receive and use that same identity. Cloud-specific location data is optional metadata, not the platform's definition of a region.

Initial spike findings

• The self-managed logical region commonly defaults to ncp.
• Region identity currently has multiple spellings and meanings:
  • The NATS chart default supplies dc:ncp in deploy/helm/nats/values.yaml.
  • The API and gRPC paths use dc placement in some configuration, including deploy/helm/grpc-proxy/grpc-proxy/values.yaml.
  • HTTP invocation creates region-scoped JetStream streams using an aws-region placement key in src/invocation-plane-services/http-invocation/crates/server/src/nats/mod.rs.
  • Environment variables such as AWS_REGION and NVCF_REGION are used in different places and cannot be treated as equivalent without checking whether they configure NVCF placement or an actual cloud SDK.
• Cassandra keyspace migrations and chart defaults assume the ncp datacenter, for example migrations/cassandra/keyspaces/nvcf_api/01_init_keyspace.up.sql. Some datacenter expansion plumbing exists, but the full replication and lifecycle contract is not defined.
• Compute-plane APIs already model cloud provider and region as separate fields. That is a useful precedent for a provider-neutral control-plane contract.
• Invocation documentation and service behavior already assume regional request paths and regional NATS placement. The missing piece is a consistent stack-wide configuration and deployment contract.

The immediate lesson from #1365 is that adding another tag unblocks one stream, but aliases will continue to drift until the region identity has one source of truth.

Configuration contract to establish

Define one canonical, provider-neutral topology contract. The exact names must be finalized with the companion reference architecture, but the contract needs to distinguish at least:

• Logical NVCF region ID: stable platform identity used for placement, routing, observability, and ownership.
• Provider: optional infrastructure implementation, such as AWS, another cloud, or on-premises.
• Provider region: optional opaque location understood by the selected provider.
• Cell or control-plane ID: identity for a deployment cell when more than one cell can exist in a logical region.

A possible values shape is:

global:
  topology:
    region:
      id: region-a
      provider: generic
      providerRegion: ""
    cell:
      id: region-a-1

Compatibility tags such as dc:<region> and aws-region:<region> should be derived at one boundary while older consumers are migrated. They should not become independent operator inputs.

Scope

• Inventory every region, datacenter, availability zone, location, and provider value across Helmfile, Helm charts, services, migrations, CLI flows, tests, and documentation.
• Classify each use as one of:
  • Logical NVCF region identity.
  • Provider-specific configuration required by a cloud SDK or service.
  • Failure-domain metadata such as datacenter or availability zone.
• Add the canonical topology values to the self-managed environment contract and validate required combinations.
• Propagate the values through nvcf-cli, Helmfile, chart values, pod environments, service configuration, migrations, hooks, and generated resources.
• Update all regional consumers, including NATS placement, invocation services, API services, routing, Cassandra datacenter and replication configuration, cluster registration, health checks, logs, traces, metrics, and operator diagnostics.
• Preserve a backward-compatible single-region default while providing deprecation guidance for legacy aliases.
• Add fail-fast validation for conflicting region settings and missing NATS or Cassandra topology.
• Support repeatable regional inventory so regions can be added, upgraded, drained, and removed without hand-editing unrelated service configuration.

Argo Rollouts compatibility

Multi-region configuration must remain correct when a service uses Argo Rollouts instead of a Kubernetes Deployment.

• Stable, canary, and preview ReplicaSets receive the same logical region and cell identity.
• Region configuration is part of the pod template so a relevant change produces a rollout.
• Promotion, abort, retry, and rollback preserve the correct NATS placement, Cassandra locality, routing, and observability attributes.
• Analysis jobs and pre/post promotion hooks receive the same regional context when they need it.
• Tests cover both regular Deployment rendering and Argo Rollout rendering.
• The deployment workflow documents rollout order, health gates, and rollback across regions. Do not assume one Argo Rollout resource coordinates multiple clusters.

Five-region managed deployment comparison

Before this work is considered a replacement for the current managed bring-up, produce and review a gap matrix against the existing global five-region NVIDIA-managed NVCF deployment. Cover at least:

• Regional service inventory and replica topology.
• Configuration inputs, defaults, and environment overlays.
• NATS topology, placement, replication, gateways, and failure behavior.
• Cassandra topology, consistency, replication, repair, and schema migration.
• Global and regional routing, ingress, DNS, and failover.
• Secret, certificate, and identity distribution.
• Argo Rollouts behavior, promotion ordering, rollback, and supported version skew.
• Observability, alerting, diagnostics, and regional health reporting.
• Region add, upgrade, drain, failover, recovery, and removal procedures.

Provider-specific details from the managed deployment should be treated as implementation inputs, not copied into the public platform contract. Any intentional difference needs a documented reason and an owner.

Definition of done

• The canonical region, provider, provider-region, and cell semantics are documented and agreed with the reference architecture work.
• Every regional configuration consumer is inventoried and either migrated or explicitly documented as provider-specific.
• The self-managed stack renders two distinct regions and a representative five-region inventory from the same provider-neutral configuration model.
• Existing single-region installations retain a documented compatibility path and default behavior.
• NATS servers and every region-scoped JetStream stream use compatible placement derived from the canonical region ID.
• Cassandra datacenter and keyspace replication settings are derived from the supported topology rather than hard-coded to ncp.
• Deployment and Argo Rollout render tests verify region propagation to stable, canary, preview, analysis, and hook workloads as applicable.
• A two-region end-to-end bring-up verifies registration, deployment, invocation, routing, observability, upgrade, rollback, and recovery behavior.
• A five-region qualification exercise or an approved qualification plan demonstrates parity with the current managed deployment before replacement is claimed.
• The managed deployment gap matrix is reviewed, and unresolved gaps have linked follow-up issues.
• Public operator documentation explains the topology values, examples, validation errors, migration path, and compatibility aliases.

Out of scope

• Replacing the short-term compatibility fix in #1365 before the canonical contract is available.
• Selecting the final NATS and Cassandra multi-region architecture. That is tracked in the companion reference-architecture issue.
• Treating multi-cluster compute registration as equivalent to multiple regional control planes.

Resources

• Parent epic: #41
• Public roadmap: #27
• Short-term NATS placement fix: #1365
• Single-region control-plane resiliency: #1063
• Pylon multi-region discovery design: #1328
• NATS supercluster topology: https://docs.nats.io/learn/topologies/super-clusters

Start with the companion reference-architecture work and inventory the named entry points: deploy/helm/nats/values.yaml, deploy/helm/grpc-proxy/grpc-proxy/values.yaml, src/invocation-plane-services/http-invocation/crates/server/src/nats/mod.rs, and migrations/cassandra/keyspaces/nvcf_api/01_init_keyspace.up.sql. Compare these uses with the managed five-region deployment; done means the canonical topology is agreed, propagated and validated across two- and five-region renders, rollouts, migrations, tests, and documentation.