[RFC]: [Store] Recoverable client liveness states and asynchronous offboarding

### Changes proposed

## Motivation

Mooncake Store currently treats one client heartbeat timeout as a permanent failure. Temporary network jitter, a client pause, or a delayed Master monitor loop can therefore irreversibly unmount recoverable resources and delete their metadata.

We propose adding a recovery window: after the first TTL expires, the Master retains the client's resources but stops routing new requests and work to them; a valid liveness signal makes them serving again. Terminal cleanup starts only if the client remains unavailable for a second TTL.

Resource availability checks are currently distributed across allocation, replica lookup, Copy/Move, task dispatch, drain, offload, and promotion. This proposal also centralizes that decision so the new liveness semantics are applied consistently.

## Changes proposed

- Introduce an **Active → Suspected → Offline** state machine for each client incarnation, with separate active and suspicion TTLs.
- Add a typed `ServingEligibility` seam for all client-owned resource routing, while retaining Suspected resources for possible recovery.
- Separate client liveness from remount readiness and segment lifecycle.
- Express terminal cleanup as a retryable `ClientOffboardingJob`; complete it synchronously first, then move the same job to an asynchronous executor.
- Preserve the existing Ping RPC schema and legacy TTL configuration.
- Rebuild process-local liveness records when restoring Master snapshots; keep NoF on its independent health model.

## Proposed design

### Client liveness

Each client incarnation has a `ClientLivenessRecord`:

```text
Active --active TTL--> Suspected --suspicion TTL--> Offline
^ |
+---- liveness signal --+
```

- **Active** resources may serve new requests.
- **Suspected** resources are retained but cannot receive new allocations, descriptors, or work. A valid liveness signal recovers the same incarnation.
- **Offline** is terminal for that incarnation and starts offboarding.

The suspicion TTL starts when the Master actually commits `Active -> Suspected`, so a delayed monitor does not shorten the recovery window. Ping and successful segment mounts are liveness signals; ordinary Put/Get traffic and task/offload/promotion heartbeats are not.

Ping remains wire-compatible: a ready Active client receives `OK`; an unknown or Offline client receives `NEED_REMOUNT`; Ping may recover a Suspected client before applying the existing remount-readiness check.

Client liveness, remount readiness (`ok_client_`), and segment lifecycle remain independent. In particular, `ok_client_` will no longer determine whether resources may serve or must be retained.

### `ServingEligibility`

Allocation, metadata reads, task routing, offload, promotion, and drain will use a small typed interface with the following common semantics:

```text
resource can serve
= resource is intrinsically valid
&& segment lifecycle permits serving
&& owning ClientLivenessRecord is Active

ShouldRetainResources
= ClientLivenessRecord is Active or Suspected
```

This supplements existing segment checks rather than replacing them. NoF resources continue to use their own probe and lifecycle rules.

While Suspected, the Master blocks new dependencies on the client, including allocations, descriptors, Copy/Move routing, task delivery, offload/promotion work, and drain selection. End, Revoke, failure, and completion callbacks for already-started work remain accepted so refcounts, quota, and task state can converge.

Memory allocator registrations and buffers retain the specific `ClientLivenessRecord` that created them, preventing a later incarnation from reactivating old memory resources.

### Offboarding

Terminal retirement uses one job interface:

```text
PrepareClientOffboarding(record) -> ClientOffboardingJob
CompleteClientOffboarding(job)
```

Prepare commits Offline and immediately isolates the incarnation. Complete performs the expensive work: metadata cleanup, memory segment unmount, LocalDisk/offload/promotion cleanup, quota convergence, and external metadata cleanup. The old record is removed only after all steps succeed; until then it acts as a Mount/ReMount barrier.

The first implementation calls Complete synchronously from the client monitor. A follow-up submits the same job to an asynchronous executor with batching, retry, shutdown draining, and queue observability. This keeps state and cleanup semantics unchanged when execution becomes asynchronous.

### HA restore and compatibility

After snapshot restore, each restored client starts as Active with a fresh active TTL, preserving the current recovery semantics. The Master rebuilds the liveness associations needed by restored resources, while readiness still follows the existing ReMount protocol.

Snapshot creation must not capture partially completed offboarding. The synchronous phase preserves the current exclusion behavior; asynchronous offboarding will add explicit snapshot/offboarding coordination.

New configuration:

```yaml
client_active_ttl_sec: 10
client_suspicion_ttl_sec: 20
```

`client_live_ttl_sec` and `--client_ttl` remain deprecated aliases for the active TTL. If an active/legacy TTL is explicitly configured but suspicion TTL is omitted, suspicion inherits the resolved active TTL. No RPC schema change is required.

LocalDisk and task metadata currently identify ownership by client ID. Initially, complete cleanup plus the Offline remount barrier prevents old resources from overlapping a new incarnation. An explicit incarnation token would provide stricter isolation for artificially reused client IDs, but requires a wire/schema change and is outside the initial scope.

## Development plan

1. Add `ClientLivenessRecord`, dual-TTL configuration, Ping/Mount integration, and the three-state monitor.
2. Introduce `ServingEligibility` across allocation, descriptors, Copy/Move, task/drain, LocalDisk/offload, and promotion.
3. Add synchronous `ClientOffboardingJob`, retry semantics, HA rebinding, metrics, and regression coverage.
4. Move job completion to the asynchronous executor without changing state semantics.

Implementation may proceed while this RFC is discussed, in reviewable commits that can be adjusted before merge.

## Open questions

1. Should Admin descriptor queries hide Suspected resources while diagnostic status queries still expose them?
2. Is it acceptable for new offload/promotion work rejected during Suspected to require a later rescan rather than automatic replay initially?
3. Should snapshots wait for pending asynchronous offboarding jobs, or should offboarding state eventually be serialized?
4. Is preserving the current RPC schema preferable to adding an explicit incarnation token in the first version?

Feedback on the state semantics, `ServingEligibility` seam, and staged synchronous-to-asynchronous offboarding plan is especially welcome.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by reading the existing client heartbeat/TTL monitor and Ping/Mount paths, then trace the named allocation, routing, offload, promotion, drain, and snapshot-restore paths. Done means the proposed liveness states, serving-eligibility seam, synchronous offboarding, compatibility rules, and regression coverage are implemented. Resolve the listed open questions before treating the design as settled.

Written by the indexing model from the issue text.