[RFC]: Fixed-Lifecycle Soft Pin with Request-Level TTL and Deadline-Indexed Expiration

## Changes proposed

### Summary

This RFC proposes changing Mooncake Store soft pinning from an access-renewed lease into a fixed-lifecycle eviction-priority hint.

A soft pin starts when the first replica of a write becomes readable. Its deadline is fixed for that activation and is not extended by `Get`, `Exist`, group lease refresh, replica copy/move, or storage promotion. When the deadline is reached, the object becomes ordinary cache. A later write may explicitly enable soft pinning again.

The proposal also adds an optional request-level TTL, an explicit `Preserve | Enable | Disable` write action, and a deadline index so expiration cleanup processes only due objects instead of scanning all metadata.

Related work:

- Implementation prototype: https://github.com/kvcache-ai/Mooncake/pull/2909
- Group-retention consumer: https://github.com/kvcache-ai/Mooncake/pull/2835

### Motivation

The current soft-pin representation retains a `soft_pin_timeout` after the deadline has passed. Read-lease refresh paths can subsequently move that timeout into the future, causing an expired soft pin to become active again simply because the object was accessed.

This behavior creates several problems:

- Soft pin duration is not fixed and can be extended indefinitely by reads.
- An object can regain soft-pin priority without an explicit write intent.
- Group reads can indirectly refresh every member's object-level soft pin.
- The soft-pin gauge can count expired metadata until another operation happens to normalize it.
- A periodic exact cleanup implemented as a full metadata scan costs `O(total object count)` even when few or no soft pins expire.
- The behavior is difficult to compose with group retention because object soft pin and group lifetime can accidentally refresh each other.

The desired behavior is a predictable priority lifetime suitable for both standalone objects and future group-retention policy.

### Proposed semantics

1. A soft pin is committed when a write first changes the object from zero readable replicas to at least one readable replica.
2. `Enable(ttl)` sets a deadline relative to that completion time.
3. Reads grant only the ordinary read lease and never extend or reactivate the soft pin.
4. At `now >= deadline`, the object is treated as ordinary cache.
5. A later explicit `Enable` may start a new soft-pin lifetime.
6. `Disable` removes the committed soft pin when the write becomes readable.
7. `Preserve` retains an unexpired committed deadline across Upsert, but does not preserve an already expired deadline.
8. `Enable` with `TTL = 0` is equivalent to committing ordinary cache.
9. Soft pin remains an eviction-priority hint. If `allow_evict_soft_pinned_objects=true`, an active soft-pinned object may still be selected by the existing second eviction pass.
10. Explicit removal APIs continue to remove soft-pinned objects.

### Write API

Replace the ambiguous boolean write intent with an explicit action:

```cpp
enum class SoftPinAction : uint8_t {
PRESERVE = 0,
ENABLE = 1,
DISABLE = 2,
};

struct ReplicateConfig {
// Existing fields omitted.
SoftPinAction soft_pin_action{SoftPinAction::PRESERVE};
std::optional soft_pin_ttl_ms{};
};
```

Request rules:

- `PRESERVE` must not include `soft_pin_ttl_ms`.
- `DISABLE` must not include `soft_pin_ttl_ms`.
- `ENABLE` uses `soft_pin_ttl_ms` when present.
- `ENABLE` without an override uses the Master's `default_kv_soft_pin_ttl`.
- A request-level TTL above `max_kv_soft_pin_ttl` is rejected.
- Unknown action values are rejected.

`PRESERVE` is the default because a default Upsert should not silently remove an existing eviction-priority deadline. For a new Put, preserving an absent deadline simply creates ordinary cache unless the caller selects `ENABLE`.

The existing C ABI can retain its boolean field for the initial change and translate it as follows:

```text
with_soft_pin=true -> ENABLE with the Master default TTL
with_soft_pin=false -> PRESERVE
```

This keeps existing Go and Rust wrappers functional, although those wrappers will not expose explicit disable or request-level TTL until their APIs are extended.

### Master configuration

Add:

```text
max_kv_soft_pin_ttl = 24h by default
```

The Master validates that:

```text
default_kv_soft_pin_ttl <= max_kv_soft_pin_ttl
```

Deadline arithmetic is checked when the soft pin is committed, not only at Master startup. If `completion_now + ttl` cannot be represented by `system_clock::time_point`, the deadline saturates at `time_point::max()` instead of overflowing.

### Object state model

The object-level state is:

```text
Disabled
Pending(action, ttl, eligible_replica_ids)
Active(deadline)
```

`Pending` is write intent and `Active` is committed eviction priority. Starting a write does not immediately apply the requested action because the object may never become readable.

The pending action is committed only when:

- the End operation completes a replica allocated or reused by the current write; and
- the object changes from zero completed replicas to at least one completed replica.

After commit, Pending is cleared. Later replica completions and duplicate End calls therefore do not refresh the deadline.

Replica IDs are used to associate Pending with the replicas of the current write. End currently carries no write-generation token, so it is not possible to distinguish every stale End from a newer same-client write. Adding a protocol-level operation token is outside this RFC.

### Revoke, preemption, and processing cleanup

A partial Revoke must not unconditionally discard Pending.

Pending is retained while at least one eligible PROCESSING replica from the same write can still become the first readable replica. Pending is cleared when:

- the object is deleted;
- a newer Upsert preempts the write; or
- no eligible and valid PROCESSING replica remains.

Committed state is independent from Pending. An Upsert that has not yet become readable does not immediately consume or replace the previous committed deadline.

For size-changing Upsert, an unexpired committed deadline is carried into the replacement metadata. It remains subject to its original deadline while the new replicas are PROCESSING. An expired deadline is not carried into replacement metadata.

### Deadline index

Add a private Master-level deadline index without changing `MetadataShard`:

```text
min_heap
latest_registration[tenant_scoped_key] = (deadline, shard_index)
```

The index has its own mutex and is ordered after metadata-shard locks in the lock hierarchy. Cleanup never holds the index mutex while acquiring a metadata-shard lock.

The index is updated when:

- an `Enable` action commits;
- an inherited active deadline is installed in replacement metadata;
- an active object is deleted;
- an operation explicitly disables or lazily expires the committed deadline; or
- restored metadata is successfully applied.

The index is derived runtime state and is not serialized in snapshots.

### Expiration cleanup

The existing task-cleanup thread remains responsible for soft-pin expiration; this RFC does not add another thread.

Each cleanup cycle:

1. Pops heap entries with `deadline <= now`.
2. Discards entries that do not match the latest registration.
3. Groups remaining entries by metadata shard.
4. Acquires each affected shard's read lock once.
5. Looks up the object and clears the soft pin only if the metadata still has the same deadline.
6. Applies the aggregate gauge decrement after metadata updates.

The exact-deadline comparison prevents an old heap entry from clearing a soft pin that was subsequently re-enabled with a newer deadline.

Expiration therefore changes from:

```text
O(total metadata objects) per cleanup cycle
```

to approximately:

```text
O(number of due entries * log(heap size))
```

with metadata locks acquired only for shards containing due entries.

### TTL updates and stale heap entries

Updating a deadline inserts a new heap entry and updates the latest-registration map. The old heap entry is lazily invalidated because `std::priority_queue` does not support efficient arbitrary deletion.

To prevent repeated TTL updates from growing the heap without bound, rebuild it from live registrations when:

```text
heap_size > max(4096, live_registration_count * 2)
```

The factor of two limits stale entries to roughly the number of live registrations for large indexes and gives an amortized `O(1)` rebuild cost per update. The floor of 4096 avoids rebuilding a small heap after every few updates while still placing an absolute bound on stale entries for small live sets. These are internal constants rather than user-visible configuration.

### Read and eviction paths

`Get`, `Exist`, and group lease refresh stop evaluating or mutating soft-pin state. They update only the ordinary read lease.

This removes an additional object-lock acquisition from read-heavy paths and ensures that reads cannot influence object soft-pin lifetime.

Eviction paths still evaluate `now < deadline` when deciding priority. Therefore an object receives no soft-pin protection after its deadline even if the periodic cleanup has not yet removed the stored deadline.

The soft-pin gauge converges within the existing cleanup interval rather than synchronously on every read. It represents normalized active metadata, not a clock-exact value at every instant between cleanup cycles.

### Metrics

Metadata state transitions return a metric delta and an optional deadline-index mutation. The Master applies index and metric changes after releasing the object's spin lock.

The gauge changes once for each logical transition:

```text
Disabled -> Active: +1
Active -> Disabled: -1
Active -> Active with a new deadline: 0
```

Object destruction remains responsible for decrementing an active committed soft pin. Index removal and metric ownership remain separate so stale heap entries cannot double-decrement the gauge.

### Snapshot and HA semantics

Snapshots continue to serialize the committed soft-pin deadline already stored in object metadata. The heap and latest-registration map are not serialized.

During restore:

1. An expired soft deadline is normalized to Disabled.
2. The object then follows the existing lease-based restore survival rule.
3. If the ordinary read lease is still valid, the object is restored as ordinary cache.
4. If both the ordinary lease and soft deadline are expired, the existing restore cleanup may discard the object.
5. After successful metadata cleanup, the deadline index is rebuilt from surviving active deadlines.

If restoring one snapshot candidate fails and the Master falls back to another candidate, resetting metadata also clears the derived deadline index. This prevents registrations from a failed candidate from leaking into the next attempt.

This RFC does not change the snapshot wire format.

### Separation from group retention

Object soft pin and group retention must remain separate priority sources:

```text
object soft pin active
group retention active
```

Eviction policy may combine those sources, but group retention must not copy or refresh a group deadline into every member's committed object soft pin. This separation allows PR #2835 to use the fixed-lifecycle behavior without reintroducing per-member deadline refresh or state duplication.

### Compatibility and rollout

This proposal assumes a coordinated stop-the-world upgrade of Master and C++/Python clients. Mixed old/new Master-client RPC compatibility is not a goal.

Compatibility implications:

- Replacing the C++/Python `with_soft_pin` field is a source-level breaking change.
- Default same-size Upsert changes from implicitly disabling soft pin to preserving an unexpired deadline.
- Size-changing Upsert becomes consistent with same-size Upsert under the explicit action model.
- The existing C ABI boolean remains available through translation to `Enable` or `Preserve`.
- Existing snapshots remain readable because the serialized object deadline representation is unchanged.

### Alternatives considered

#### Continue lazy expiration on access

This avoids a background index but leaves metrics and metadata stale indefinitely for cold objects and ties cleanup behavior to read traffic.

#### Periodically scan all metadata

This is simple and exact at the sweep interval, but its cost scales with total object count rather than the number of active or expiring soft pins.

#### Put one heap in every metadata shard

This avoids a global index mutex but changes `MetadataShard`, duplicates heap management across 1024 shards, and requires every cleanup cycle to inspect many independent heaps. It is more invasive than needed for the initial implementation.

#### Use a timing wheel or indexed mutable heap

These structures can reduce stale entries or update cost, but introduce substantially more implementation and synchronization complexity. A min-heap plus latest-registration map is sufficient for the expected TTL granularity and update rate.

### Non-goals

- Implementing group retention itself.
- Making soft pin an absolute no-eviction guarantee.
- Changing ordinary read-lease semantics.
- Changing Copy, Move, or promotion operations into soft-pin activation points.
- Adding a new background thread or per-shard deadline structures.
- Supporting rolling mixed-version Master-client RPC upgrades.
- Adding a write-generation token to PutEnd or UpsertEnd.

### Feedback requested

The RFC especially requests feedback on:

- `PRESERVE` as the default write action;
- the source compatibility impact of replacing `with_soft_pin` in C++ and Python;
- bounded-delay gauge convergence versus read-path normalization;
- the global index and lazy-invalidation compaction policy; and
- keeping group retention as a separate eviction-priority source.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start with implementation prototype PR #2909 and compare it with the proposed soft-pinning behavior at Get, Exist, End, Revoke, cleanup, and snapshot restore entry points. Done means the explicit actions, fixed deadlines, deadline index, expiration cleanup, metrics, and restore semantics match this RFC without reintroducing read-based renewal.

Written by the indexing model from the issue text.