[RFC]: TENT Versioned Configuration Lifecycle and Staged Runtime Application

### Changes proposed

## Summary

The TENT roadmap calls for **hot-reload or staged application of configuration changes** (#1058), but the current runtime has no configuration lifecycle contract that makes such updates safe.

This RFC proposes a TENT-specific configuration model with:

- a conservative split between immutable bootstrap configuration and versioned runtime configuration;
- validated, immutable whole-configuration snapshots;
- an explicit staged `prepare -> publish -> commit/rollback` update protocol;
- well-defined consistency boundaries for in-flight requests and batches;
- explicit ownership rules for process-wide resources and multiple `TransferEngine` instances.

The first milestones establish the lifecycle and validation foundation. They do **not** enable arbitrary hot reload or change current runtime behavior.

## Motivation

TENT already supports loading configuration from `MC_TENT_CONF`, environment overrides, and caller-provided `Config` values. However, configuration is currently a mutable JSON object whose individual `get()` and `set()` operations are protected by a mutex. That prevents a data race on the JSON object, but it does not provide an atomic, coherent runtime update.

During `TransferEngineImpl::construct()`, many values are copied into engine members or used to construct long-lived components:

- RPC address, port, and server thread count;
- progress worker and admission queue limits;
- topology and platform loader;
- transport instances and transport selector;
- metrics server configuration;
- failover and request-merging behavior.

Changing the underlying `Config` after construction therefore has inconsistent effects: some code may read a new value, while already-constructed components continue using old values.

There is also process-wide state:

- `Platform::getLoader()` is initialized with `std::call_once` and silently retains the first engine's configuration;
- `TentMetrics` is a process-wide singleton;
- logging level and several allocator/prober facilities are process-wide.

This makes staged updates and multiple engine instances ambiguous unless ownership and conflict behavior are defined first.

## Goals

1. Define one canonical, validated configuration representation after file/environment/API inputs are merged.
2. Classify fields conservatively as bootstrap-only, runtime-updatable, or unsupported for live update.
3. Publish runtime configuration as immutable snapshots with monotonically increasing generations.
4. Validate complete candidate snapshots, including cross-field and build-capability constraints.
5. Prevent partial visibility: readers observe either the old generation or the new generation.
6. Define safe update boundaries for requests, batches, workers, selectors, and transports.
7. Define `prepare`, `commit`, rejection, and rollback behavior for runtime consumers.
8. Make process-wide versus engine-owned resources explicit, including multi-instance conflict detection.
9. Preserve existing configuration keys, files, environment variables, and startup behavior during migration.
10. Expose bounded diagnostics for current generation, rejected updates, and restart-required fields.

## Non-goals

- Building a remote administration service or replacing the operator CLI proposed in #2864.
- Redefining the QoS Contract schema or effective policy model from #2856.
- Redefining the Backend Capability or Execution Plan model from #2863.
- Making every existing field hot-reloadable in the first implementation.
- Recreating transports, GPU contexts, RPC listeners, or metadata backends live in v1.
- Changing existing configuration precedence or silently changing defaults.
- Introducing per-request configuration mutation.

## Proposed lifecycle model

### 1. Canonical configuration bundle

After compatibility inputs are loaded and normalized, the runtime owns only two authoritative configuration objects:

```cpp
struct TentConfigBundle {
BootstrapConfig bootstrap;
std::shared_ptr runtime;
};
```

Raw JSON, environment variables, legacy flat keys, and caller overrides are inputs to the loader. They are not independent runtime sources of truth.

### 2. Conservative field classification

The exact field inventory should be completed in PR1. The initial policy should be conservative:

| Class | Examples | Update policy |
| --- | --- | --- |
| Bootstrap | platform/device plugin loading, metadata backend, RPC bind address/port/thread count, transport enablement, device topology, metrics bind address/port | restart required |
| Runtime candidate | admission limits, dispatch windows, selector policy snapshot, failover limits, degradation thresholds, bounded reporting intervals | staged update after the consumer becomes generation-aware |
| Derived/read-only | discovered topology, resolved capabilities, bound ports, loaded backend list | never set directly; recomputed or reported |
| Unknown/unsupported | unclassified keys and fields without an update contract | reject live update |

A field being placed in `RuntimeConfig` does not mean it is immediately hot-reloadable. Each consumer must explicitly declare an apply boundary and rollback behavior.

### 3. Immutable versioned snapshots

Runtime readers obtain an owning reference:

```cpp
struct RuntimeConfigSnapshot {
uint64_t generation;
RuntimeConfig config;
};

class RuntimeConfigProvider {
public:
std::shared_ptr snapshot() const;
Status stage(const RuntimeConfigCandidate&, ConfigApplyPlan&);
Status commit(ConfigApplyPlan&);
};
```

Operations that need a coherent set of values pin one snapshot at a defined boundary. A long-running batch must not mix values from multiple generations unless that behavior is explicitly documented.

### 4. Staged update protocol

A candidate update follows this pipeline:

1. Load and merge inputs using existing precedence rules.
2. Normalize aliases into canonical paths.
3. Validate types, ranges, cross-field invariants, and build capabilities.
4. Diff against the active snapshot and classify every change.
5. Reject bootstrap-only or unsupported changes with `restart_required` diagnostics.
6. Ask affected consumers to `prepare` without publishing the candidate.
7. If every consumer prepares successfully, atomically publish the new generation.
8. Commit prepared consumer state.
9. If preparation fails, discard the candidate and retain the previous generation.

The initial implementation should support only consumers for which preparation is side-effect-free or rollback is well defined.

### 5. Consistency boundaries

Suggested defaults:

- a new logical request pins the current generation;
- all slices, retries, and staging hops derived from that request retain the pinned generation;
- an admission/progress loop takes one snapshot per dispatch iteration;
- background reporters may observe the latest generation per reporting interval;
- transport construction and registered-memory ownership remain bootstrap-scoped in v1.

### 6. Process-wide and multi-instance ownership

Process-wide components must not silently use whichever engine initializes first.

The implementation should choose and document one of these patterns per component:

- move ownership into an explicit shared `TentRuntimeContext`;
- make the component engine-owned;
- keep it process-wide but validate a canonical bootstrap fingerprint and reject conflicting engine construction.

This applies at least to Platform, metrics endpoint ownership, logging configuration, shared RPC executors, probers/plugins, and global allocators. Compatible multi-engine use should remain possible; incompatible configurations should fail with actionable diagnostics.

## Compatibility

- Existing `Config`, `MC_TENT_CONF`, and environment override behavior remain supported as compatibility inputs.
- Existing applications that never perform a runtime update see no behavior change.
- Bootstrap-only changes are reported as restart-required rather than partially applied.
- Unknown fields are not silently accepted by the live-update path.
- Existing public C++/C/Python APIs remain source-compatible during the foundation phases.

## Proposed PR sequence

### PR1: Field inventory, lifecycle types, and immutable snapshot foundation

- Inventory current TENT keys and their consumers.
- Add bootstrap/runtime/derived classification metadata.
- Add typed diagnostics and immutable generation snapshots.
- Add compatibility loading into the canonical bundle.
- Add unit tests for classification, precedence, and snapshot coherence.
- No live update entry point and no data-path behavior change.

### PR2: Central validation and diff/apply planning

- Add canonical path validation and cross-field checks.
- Add build/backend capability validation.
- Produce a structured diff with `runtime_candidate`, `restart_required`, and `unsupported` results.
- Make the result reusable by future `tent check-config` work in #2864.

### PR3: First generation-aware runtime consumers

- Convert a small, low-risk subset such as admission dispatch limits or reporting intervals.
- Pin snapshots at explicit operation/loop boundaries.
- Add concurrency tests proving readers never observe mixed generations.

### PR4: Staged prepare/commit/rollback coordinator

- Register runtime consumers and affected paths.
- Prepare all consumers before atomic publication.
- Add failure injection and rollback tests.
- Emit bounded generation/update/rejection metrics and logs.

### PR5: Local update entry point

- Add a bounded local API or file-reload trigger after the lifecycle is proven.
- Keep remote administration outside this RFC.
- Reuse validation and diagnostics rather than creating a second config parser.

### PR6: Multi-instance ownership and conflict enforcement

- Introduce an explicit runtime context or bootstrap fingerprint validation for process-wide resources.
- Add create/destroy/recreate and two-engine tests.
- Reject incompatible process-wide settings instead of silently accepting first-writer-wins behavior.

## Validation plan

- Golden tests for current configuration precedence and legacy keys.
- Complete field-classification coverage: every accepted key has one lifecycle class.
- Cross-field and build-capability validation tests.
- Concurrent readers during successful and rejected updates.
- Failure injection at every prepare step.
- Proof that a rejected candidate leaves generation and consumer state unchanged.
- In-flight batch tests across a generation change.
- Multiple-engine tests with compatible and conflicting bootstrap configurations.
- Sanitized diagnostics with no secrets, raw addresses, rkeys, or unbounded labels.
- Performance checks showing snapshot access adds no meaningful per-slice overhead.

## Open questions

1. Should the first public update source be an explicit local API, file reload, or both?
2. Which runtime fields are safe enough for the first generation-aware consumer PR?
3. Should one process own a single shared `TentRuntimeContext`, or should process-wide facilities be minimized over time?
4. Must commit be infallible after successful prepare, or do any consumers require compensating rollback?
5. Should long-running batches always pin one generation, or may selected telemetry-only fields update between retries?

## Relationship to existing work

- #1058 directly lists hot-reload or staged application of TENT configuration changes.
- #1883 is the overall Mooncake roadmap and calls for production-ready, modular TENT behavior.
- #2864 defines operator CLI and configuration diagnostics, but explicitly keeps hot reload/remote administration out of scope. This RFC supplies the lifecycle model that future `check-config` restart-required diagnostics can consume.
- #2694 is relevant prior art for bootstrap/runtime configuration ownership, but it is scoped to Mooncake Store Master. This RFC is TENT-specific and includes request/batch consistency, transport lifecycle, and multi-engine process ownership.
- #2856 defines the QoS Contract schema and effective policy model. Runtime configuration may carry a validated contract snapshot, but this RFC does not redefine its semantics.
- #2863 defines backend capabilities and Execution Plans. Configuration generations may later be an input to plan construction/explain output, but this RFC does not redefine plans.
- #3550 addresses logging backend standardization. This RFC only defines process-wide configuration ownership/conflict semantics and does not choose a logging implementation.

## Before submitting

I searched existing issues and PRs for TENT configuration, hot reload, staged application, runtime configuration, bootstrap configuration, multi-instance lifecycle, and process-wide ownership. I did not find an RFC that defines this TENT-specific lifecycle contract.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by tracing Config loading from MC_TENT_CONF, environment overrides, and caller-provided values into TransferEngineImpl::construct(). Review the proposed PR1 scope: field inventory, lifecycle classification, canonical loading, immutable generation snapshots, and tests for precedence and snapshot coherence. Done for that phase means the foundation is covered without adding a live-update entry point or changing runtime behavior.

Written by the indexing model from the issue text.