[RFC]: Reliable Metadata Across Node Replacement

### Changes proposed

## Summary

A Transfer Engine process may restart while keeping the same segment name.
Remote nodes can then retain metadata and RDMA resources that belong to the old
process.

We propose a small recovery protocol:

1. Refresh cached remote metadata after a bounded TTL.
2. Give every process incarnation a random `instance_id`.
3. Treat a segment descriptor as one immutable snapshot.
4. When `instance_id` changes, retire all resources derived from the old peer.
5. On transport failure, force a refresh instead of waiting for TTL.

The work is split into four stages so that cache behavior, wire compatibility,
and RDMA recovery can be reviewed independently.

## Problem

A segment name identifies a logical endpoint, not a particular process. After a
replacement, the new process may publish different:

- NIC addresses and device attributes;
- memory addresses and registration keys;
- topology and routing information.

Comparing those fields is useful, but it is not a reliable replacement signal.
A process restarted on the same host may reuse NIC identity, virtual addresses,
or registration key values.

A version counter alone also does not solve the problem. If every process starts
at version 1, two different instances are indistinguishable. Seeding the counter
from wall-clock time adds clock assumptions without providing a strict identity.

The required property is:

> A consumer eventually detects that a segment name is owned by a new process,
> stops creating work from the old snapshot, retires old derived resources, and
> either recovers on the new process or returns a bounded failure.

## Scope and Assumptions

This RFC assumes:

- one legitimate publisher per segment name at a time;
- atomic read and write of one metadata key;
- a complete segment descriptor is stored under that key;
- RDMA endpoints remain alive until their outstanding work drains;
- metadata requests and RDMA operations can fail.

This RFC covers remote metadata caching and RDMA recovery after process
replacement. It does not cover:

- two live publishers competing for the same segment name;
- ordering concurrent writers within one process;
- safe memory deregistration while remote work is in flight;
- metadata-service replication or disaster recovery.

If exclusive ownership cannot be guaranteed, a lease and fencing protocol is
required in addition to this design.

## Design

### Process incarnation

Add an opaque process identity to `SegmentDesc`:

```cpp
struct SegmentDesc {
std::string instance_id;
// Existing descriptor fields.
};
```

`instance_id` must:

- be generated once when Transfer Engine starts;
- contain at least 128 random bits;
- remain unchanged for the process lifetime;
- change on every process start;
- have no ordering or wall-clock semantics.

All segments published by one Transfer Engine instance use the same ID.

### Immutable snapshot

The descriptor, including `instance_id`, devices, and buffers, is published as
one metadata value. A reader installs it only after the complete value has been
fetched, decoded, and validated:

```text
fetch -> decode -> validate -> atomically replace cached shared_ptr
```

A failed or malformed refresh must not overwrite a valid snapshot. Network I/O
must not run while holding `segment_lock_`.

### Bounded refresh

Each remote cache entry records its last successful refresh:

```cpp
struct SegmentCacheEntry {
std::shared_ptr snapshot;
uint64_t refreshed_at_ns;
};
```

Lookup behavior is:

| Condition | Result |
|---|---|
| Local segment | Return the local snapshot |
| Remote snapshot within TTL | Return the cached snapshot |
| Remote snapshot past TTL | Fetch and install a new snapshot |
| Ordinary refresh fails | Keep and return the old snapshot |
| Forced refresh | Bypass TTL |
| Forced refresh fails | Return failure, not stale success |

TTL controls when replacement is discovered during normal operation. It is not
the replacement identity.

### Replacement handling

The RDMA worker tracks minimal state per target:

```cpp
struct TargetPeerState {
std::string instance_id;
std::vector peer_nic_paths;
};
```

When a newly observed `instance_id` differs from the recorded value:

1. update the target state to the new instance;
2. retire every endpoint associated with the old instance;
3. clear rail failure and pause state for its paths;
4. rebuild queued slices from the new descriptor;
5. establish new endpoints lazily.

Endpoint deletion must occur after releasing the target-state mutex. Retired
endpoints continue through the existing two-phase destruction path so in-flight
work can drain safely.

The first implementation invalidates the complete target. Field-level
invalidation is an optional optimization for later.

### Failure-triggered recovery

TTL leaves an observation window between replacement and the next refresh.
Recoverable handshake, QP, CQ, or work-request failures therefore trigger:

```text
retire failed endpoint
-> force metadata refresh
-> detect replacement
-> reselect buffer, device, rkey, and NIC path
-> retry with the existing bounded budget
```

A forced-refresh failure consumes the retry budget or fails the operation. It
must not create an unbounded retry loop.

## Delivery Plan

### Stage 1: Bounded metadata cache

**Goal:** ensure remote descriptors do not remain cached forever.

Changes:

- add a successful-refresh timestamp per remote cache entry;
- apply `metadata_cache_ttl_ms` to remote lookups;
- preserve local-segment lookup behavior;
- fetch metadata without holding the cache lock;
- define ordinary and forced-refresh failure behavior.

Exit criteria:

- TTL and forced-refresh tests pass;
- malformed or failed refreshes cannot replace a valid snapshot;
- concurrent readers always observe a complete descriptor.

This stage does not change the wire format or invalidate RDMA resources.

### Stage 2: Incarnation identity

**Goal:** distinguish two processes that publish the same segment name.

Changes:

- generate one random `instance_id` at process startup;
- add backward-compatible JSON encoding and decoding;
- publish the ID in every local segment descriptor;
- treat a missing ID as a legacy peer rather than synthesizing one.

Exit criteria:

- two restarts always produce different IDs;
- old descriptors remain decodable;
- new descriptors can be read by older implementations that ignore unknown
fields.

This stage exposes identity but does not yet act on replacement.

### Stage 3: RDMA replacement recovery

**Goal:** stop using resources derived from a replaced peer.

Changes:

- track `instance_id` and paths per target in `WorkerPool`;
- detect incarnation changes after metadata lookup;
- retire old endpoints and clear old rail state;
- rebuild queued slices from the new descriptor;
- force refresh from recoverable transport-failure paths.

Exit criteria:

- replacement is detected even when descriptor resource fields are reused;
- old endpoints follow two-phase destruction;
- the new peer can establish an endpoint and complete the transfer;
- refresh and retry behavior remains bounded.

### Stage 4: Hardening and rollout

**Goal:** make replacement recovery observable and safe to deploy gradually.

Changes:

- add replacement, refresh, endpoint-retirement, and retry-exhaustion logs;
- add refresh and replacement counters;
- run mixed-version compatibility tests;
- add an end-to-end restart test;
- document TTL semantics and select a production default.

Exit criteria:

- old and new nodes can coexist during rollout;
- operators can identify refresh failures and replacement recovery;
- the restart test passes without endpoint leaks, deadlocks, or infinite
retries.

## Compatibility

Readers accept descriptors with or without `instance_id`.

For a legacy peer without an ID:

- TTL refresh still updates descriptor contents;
- transport failure still retires the failed endpoint and forces a refresh;
- proactive and unambiguous replacement detection is unavailable.

Rollout order is:

1. deploy readers that accept both formats;
2. enable writers to publish `instance_id`;
3. enable RDMA invalidation on ID changes;
4. remove legacy behavior only after the compatibility window closes.

## Validation

The essential scenarios are:

1. A cached descriptor is refreshed after TTL but not before it.
2. A transport failure bypasses TTL.
3. A new `instance_id` retires old endpoints and rail state.
4. The same `instance_id` does not cause replacement cleanup.
5. Replacement is detected when NIC names, addresses, and simulated rkeys are
unchanged.
6. A failed refresh or retry returns a bounded failure.
7. Concurrent lookup and replacement do not expose partial snapshots or create
lock-order deadlocks.
8. Old descriptors remain usable during rolling upgrade.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by locating SegmentDesc serialization, remote metadata cache lookup, and WorkerPool target-state and transport-failure paths. Review the TTL, forced-refresh, instance-change, compatibility, and restart scenarios in the validation plan. Done means bounded refresh behavior, safe snapshot replacement, old-endpoint retirement, mixed-version compatibility, and restart recovery without leaks or deadlocks.

Written by the indexing model from the issue text.