NVCF Control Plane Resiliency

Is this related to a problem? Please describe.

NVCF self-hosted runs one control plane, with every component as a single replica.

• One node failure is a full outage. Everything stops until the pod is rescheduled, which takes minutes.
• In-flight work is lost. Running invocations fail, and NVCA agents lose their connection to the control plane.
• The singleton services have no failover path — not the NATS JetStream leader, the OpenBao active node, or the Cassandra coordinator.
• There is no recovery contract. With no stated recovery time, an operator cannot commit to an SLA.
• Every upgrade needs downtime. A single-replica rollout cannot roll.

Describe the solution you'd like

Make one control plane survive failure within the cluster it runs on. Stateless services run as several replicas behind a load balancer, stateful services as clustered groups that keep working when a member is lost, and replicas are placed so no one failure can take them all. Covering:

• NVCF keeps serving through any failure inside the control plane, without operator intervention.
• No committed data is lost when something fails.
• Failover is invisible to users. They do not see errors while the platform recovers.
• Maintenance and upgrades do not require downtime.
• An operator can turn resilience on, and adjust it, through a single setting, without downtime or rebuilding an existing deployment.
• A recovery time is published per failure mode and met.
• The operator is warned when the platform can no longer absorb a failure.
• The supported topology, its infrastructure prerequisites, failure modes, and runbooks are published, and validated before release.
• LLM routing, streaming, and callback invocation paths survive component failure, with no dropped invocations.

Additional context

Related open issues: #41 (Support multi-region NVCF control planes), #44 (Support customer-managed infrastructure dependencies)

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.

No files, tests, or entry points are named. Start by mapping the self-hosted control-plane deployment and reviewing related issues #41 and #44, then define supported replicas, failure modes, prerequisites, and runbooks. Done means the topology survives stated failures without operator intervention or data loss, supports downtime-free changes, and is validated before release.