[RFC]: [Store] Lease Recovery Semantics, Steady Runtime Clock, and Snapshot Clock Mapping

### Changes proposed

## Summary

This RFC defines the long-term lease and recovery model for Mooncake Store. It addresses three related problems while keeping two existing persistence systems completely independent:

1. The standby snapshot/oplog path must not persist frequently changing lease deadlines. After promotion, an object remains protected while its memory segment is not readable, and receives a new recovery lease only when the segment successfully remounts.
2. Runtime object leases and soft-pin deadlines must use `std::chrono::steady_clock`, so normal lease decisions are not affected by NTP corrections or manual wall-clock changes.
3. The complete MasterService snapshot must preserve historical lease and soft-pin deadlines. It will persist steady-clock deadlines together with one system-clock/steady-clock mapping, allowing a new Master to reconstruct deadlines in its own steady-clock domain while still consuming downtime and restore time.

The implementation is deliberately split into three ordered pull requests:

| Phase | Pull request | Main result |
| --- | --- | --- |
| 1 | Standby snapshot/oplog recovery lease | No persisted lease deadlines; unreadable memory replicas are not evictable; remount grants recovery leases |
| 2 | Runtime steady clock | All runtime object lease comparisons use `steady_clock` |
| 3 | MasterService snapshot clock mapping | Complete snapshots persist steady deadlines and a versioned clock mapping, with v1 compatibility |

The standby snapshot/oplog path and the complete MasterService snapshot path remain parallel systems. They do not convert into one another, inherit one another's wire format, or switch at runtime.

## Motivation

### Current runtime model

`ObjectMetadata` currently represents object retention with:

- an object-level hard lease deadline;
- an optional soft-pin deadline;
- an immutable `hard_pinned` flag;
- replica status, reference count, and memory-handle validity.

Normal reads call the existing lease-grant helper. Eviction, cleanup, and snapshot restore compare the stored deadlines. The current implementation uses `std::chrono::system_clock::time_point` for both hard lease and soft-pin deadlines.

The main default values are:

- `default_kv_lease_ttl`: 10 seconds by default;
- `default_kv_soft_pin_ttl`: 30 minutes by default;
- hard pin and soft pin disabled unless requested by the object or operation;
- `allow_evict_soft_pinned_objects`: an existing policy switch for memory pressure.

### Why standby recovery cannot reuse persisted lease deadlines

The new standby snapshot/oplog path is intended to keep a hot standby close to the primary and then promote it. Persisting every read-driven lease renewal would create a high-volume, cache-policy oplog and would make a transient retention policy unnecessarily durable.

More importantly, promotion can restore tens of millions or even nearly one hundred million objects per segment. The primary can be declared before every client has remounted its segment, but the objects cannot serve reads until their memory buffers are restored. If a lease is assigned at the beginning of recovery, the lease may expire before the segment becomes serviceable. That would expose a much shorter or zero usable window to clients.

An unreadable memory replica also provides no useful memory-reclamation benefit. Evicting it before remount does not recover serviceable memory and can discard the only recoverable copy.

### Why complete MasterService snapshots are different

A complete MasterService snapshot represents a consistent running Master state. Its lease and soft-pin deadlines are part of that state and must retain their historical meaning. Resetting all snapshot objects to a default recovery TTL would extend expired objects and change cache-retention semantics.

The complete snapshot path therefore needs a clock bridge rather than a recovery reset. The bridge must account for the time between snapshot creation and restore, including time spent decoding and applying the snapshot.

## Goals

1. Use `steady_clock` for all runtime object lease and soft-pin deadline comparisons.
2. Keep lease and soft-pin deadlines out of the standby snapshot/oplog wire format.
3. Persist `hard_pinned` and `soft_pin_enabled` in standby metadata.
4. Allow a promoted Master to announce primary identity before all segments are serviceable.
5. Keep objects on an unremounted or unreadable memory segment out of ordinary memory eviction indefinitely, without adding a check to the normal read hot path.
6. Refresh every object associated with a successfully remounted memory segment, regardless of whether another readable replica already existed.
7. Make recovery TTL independently configurable, defaulting to the normal read lease TTL.
8. Preserve hard pin and soft-pin enabled state through recovery.
9. Preserve complete MasterService snapshot lease semantics across process and machine restart.
10. Read the current v1 complete snapshot format after the new implementation is deployed.
11. Reject a complete snapshot with an invalid clock mapping and fall back to an older candidate rather than guessing a default deadline.
12. Keep the normal read path free of segment-recovery state checks.

## Non-goals

1. This RFC does not define recovery or eviction policy for `DISK`, `LOCAL_DISK`, or `NOF_SSD` replicas. Disk eviction remains an independent workflow.
2. This RFC does not add segment state to heartbeat messages. A future heartbeat extension may handle a live client whose segment never remounts.
3. This RFC does not introduce lease-renewal oplog records for reads.
4. This RFC does not define a segment-level effective lease or a first-read activation protocol.
5. This RFC does not add a global recovery flag or a segment lookup to every read.
6. The two snapshot systems are not converted, handed off, or switched at runtime.
7. Non-lease times such as `put_start_time`, task deadlines, and discarded-replica cleanup deadlines are not mechanically migrated by this RFC; each requires an independent audit.
8. The Master does not implement NTP synchronization, remote time arbitration, or a maximum clock-skew policy.

## Design constraints and accepted trade-offs

- Most objects have a single replica. A memory remount therefore usually makes almost every associated object readable.
- Failover is rare compared with normal reads. An O(N) remount scan and batch assignment cost is acceptable; an extra branch on every read is not.
- Some data loss is acceptable in the system, but mass eviction of replicas that cannot currently serve reads is not useful and must be prevented.
- The recovery TTL may be configured by the deployer. A default of 10 seconds is expected to leave approximately 8-9 seconds visible after a successful remount on target hardware, but the implementation must measure this rather than assume it.
- Hard pin is retained. Its exact authority over complete-snapshot cleanup is implemented as described below and remains open for maintainer review.
- Soft-pin enabled state is retained. If a future wire-format conflict makes this impossible, it must be reviewed as a separate compatibility decision; it is not removed speculatively.

## Terminology

**Readable memory replica** means a memory replica that has completed status, a valid memory handle, a valid segment endpoint, and passes the existing `IsReplicaReadable()` check.

**Recovery lease** means the lease assigned to objects after their memory segment has successfully remounted in the standby/oplog recovery path. It is not a persisted historical deadline.

**Complete snapshot** means a successfully published MasterService snapshot produced by the primary snapshot path. It is distinct from the standby snapshot used by the new oplog/standby path.

**Snapshot clock mapping** means the pair of clock readings captured once for a complete snapshot: one system-clock value and one steady-clock value from the same snapshot context.

## Core invariants

The following invariants must hold after all three phases are complete:

1. Runtime `lease_timeout` and `soft_pin_timeout` are local `steady_clock` time points.
2. A raw steady-clock value is never compared directly between processes or machines.
3. Standby snapshot/oplog data never contains a lease deadline, soft-pin deadline, lease remaining TTL, or read-renewal timestamp.
4. A recovery lease is assigned only after every buffer and allocator operation for the remounted segment has succeeded.
5. An unreadable memory replica is not an ordinary memory-eviction candidate, regardless of its object lease expiry.
6. Explicit user removal, client offboarding, stale-handle cleanup, and segment unmount cleanup remain explicit lifecycle operations and are not silently converted into ordinary eviction.
7. Complete snapshot restore never replaces a historical expired deadline with a default recovery TTL.
8. A malformed or semantically inconsistent complete snapshot candidate fails as a whole and may fall back to an older candidate.
9. `hard_pinned` remains authoritative for lease-expiry cleanup, but not for corrupt payloads, invalid descriptors, or unrecoverable in-progress state.

## Phase 1: Standby Snapshot and OpLog Recovery Lease

Phase 1 intentionally keeps the current `system_clock` runtime deadline type. It establishes the final recovery, visibility, and eviction semantics first. Phase 2 then changes the runtime clock type without changing those semantics.

### Scope and relation to PR #3141

PR #3141 contains several useful, independent fixes: offset-allocator recovery-gap release, avoidance of false memory-eviction triggers, and OpLog backpressure handling. Those changes should remain available.

Its lease-related recovery behavior is transitional for this RFC:

- the current remount implementation tracks `was_readable` in `unordered_map`;
- the current standby restore path grants a temporary normal lease so an unreadable replica becomes evictable after the TTL.

Phase 1 replaces both behaviors. The test that asserts an unremounted replica becomes evictable after the temporary lease expires must be removed or rewritten. It must not become a long-term regression contract.

### Standby metadata wire model

The standby snapshot and OpLog payload contain only state needed to rebuild object metadata and replicas:

```text
tenant_id
key
client_id
size
replica descriptors
data_type
group_id
hard_pinned
soft_pin_enabled
last_sequence_id
```

They do not contain:

```text
lease_timeout
soft_pin_timeout
lease_remaining_ttl
last_read_time
lease-renewal OpLog entries
```

`PUT_END` remains the authoritative durable descriptor update. A standby replay must not inherit a historical lease deadline from an older in-memory metadata entry.

The standby wire version is independent from the complete MasterService snapshot version. It must not start consuming the complete snapshot's clock mapping.

### Promotion and serving visibility

Promotion has two distinct milestones:

1. The Master control plane may announce primary identity.
2. Each memory segment becomes serviceable only after its client has remounted it successfully.

Before remount, the restore path may create object metadata and dummy allocator records, but:

- the memory replica remains unreadable;
- the endpoint remains invalid for serving;
- reads do not return that replica;
- ordinary memory eviction cannot select it.

No global `recovery_in_progress` flag is added to the read path. Existing replica-handle and invalid-endpoint checks remain the source of truth for readability.

### Eviction eligibility guard

The shared memory eviction predicate becomes conceptually:

```cpp
bool IsEvictableMemoryReplica(const Replica& replica) {
return replica.is_memory_replica() &&
replica.is_completed() &&
replica.get_refcnt() == 0 &&
IsReplicaReadable(replica);
}
```

The exact helper signature may follow local code conventions, but the same predicate must be used for every memory eviction decision:

1. Batch eviction candidate census;
2. Batch eviction re-validation;
3. actual replica pop or `mark_removed`;
4. OpLog remaining-descriptor prediction;
5. offload-on-evict source selection;
6. tenant quota eviction;
7. grouped-object member eviction.

The current code has several repeated lambdas. Phase 1 should either centralize them or add tests that prove all copies have identical behavior. Updating only the candidate census is insufficient: an unreadable replica must not be marked removed during the mutation phase either.

This guard has no TTL. A live client whose segment never remounts may retain metadata indefinitely under the current design. A future heartbeat carrying a segment inventory can add a separate terminal cleanup policy.

Explicit removal and lifecycle cleanup remain separate operations. The guard is not a general prohibition against deleting an unreadable replica when the user explicitly requests removal or a segment is being unmounted.

### Remount collection and unconditional refresh

`ReMountSegment()` scans metadata to find replicas whose descriptors belong to the remounted segments. It should collect a vector of object pointers:

```cpp
std::vector restored_objects;
```

The scan can deduplicate without a hash table: for each `ObjectMetadata`, set a local `matched` flag while visiting its replicas and append the metadata pointer once if any replica matches a remounted segment.

The implementation must not record `was_readable`. If a remounted segment owns an object, that object receives a recovery lease even when another memory replica was already readable. This is the chosen semantics: the remounted segment itself has just become serviceable, and the cost of a redundant lease refresh is preferable to another transition-state branch.

### Remount ordering

The operation must keep the existing exclusive snapshot lock and complete the following order:

```text
validate remount request
-> locate matching replicas
-> restore allocators and buffers
-> replace all memory buffers
-> verify the replacement succeeded
-> compute one recovery deadline for the batch
-> assign that deadline to all collected objects
-> clear invalid endpoint bookkeeping
-> publish the segment/client as serviceable
-> release the exclusive lock
```

If any allocator or buffer restoration fails:

- no recovery lease is granted for that failed remount;
- the segment is not published as serviceable;
- invalid endpoint state remains in place;
- temporary allocator and buffer state is rolled back through existing cleanup.

If a parallel lease-assignment worker fails after some objects have been updated, the remount is treated as failed for publication. Exact rollback of extended deadlines is not required because the partial update only extends protection; it cannot cause premature eviction. The failure must be observable and the segment must not be advertised as ready.

### Batch deadline assignment

The recovery deadline is calculated once per remount batch, not once per object:

```text
grant_start = current runtime lease clock now
recovery_deadline = grant_start + effective_recovery_ttl
soft_pin_deadline = grant_start + default_soft_pin_ttl
```

The current runtime lease clock is `system_clock` in Phase 1 and `steady_clock` after Phase 2.

Each object receives the same `recovery_deadline` unless its existing deadline is later; all grant helpers retain max-update semantics.

The deadline is calculated after the allocator and buffer work has completed. Consequently, allocator reconstruction time does not consume the recovery TTL. The batch assignment itself consumes some TTL before the segment is published; this visible-window cost is measured and bounded by the performance acceptance criteria below.

### Recovery TTL configuration

Add one optional configuration value:

```text
recovery_kv_lease_ttl
```

Its effective value is the configured duration or, when unset, `default_kv_lease_ttl`. It applies only to standby/oplog promotion remounts. The soft-pin refresh continues to use `default_kv_soft_pin_ttl`; no separate recovery soft-pin configuration is introduced in this RFC.

The configuration is intentionally independent so operators can choose a larger recovery window for very large segments without changing normal read retention.

### Pin state

Standby/oplog recovery preserves:

- `hard_pinned` as an immutable object property;
- `soft_pin_enabled` as an explicit property, not merely "the current soft deadline has not expired."

Before remount, soft-pin enabled objects are protected by the unreadable-replica eviction guard rather than by a persisted soft deadline. After remount, they receive the full soft-pin TTL. Hard-pinned objects remain protected independently of lease expiry.

### Cost and visible lease window

The remount path is intentionally O(N) over objects associated with a remounted segment. A preliminary local microbenchmark for a tight loop that assigns one shared deadline suggests roughly 1.4-1.9 seconds for one hundred million objects, while a production path including metadata access and locking may take approximately 2-5 seconds. Calling the runtime clock once per object is estimated to be materially slower and is prohibited.

The default 10-second recovery TTL is accepted only if the target deployment keeps approximately 8 seconds or more visible after the segment is published. Phase 1 must record:

- matched object count;
- deadline-assignment duration;
- total remount duration;
- effective recovery TTL;
- estimated visible remaining TTL at publication.

If the assignment phase exceeds the budget on target hardware, the implementation must first use bounded shard/vector parallelism. If the measured workload still cannot meet the window, the deployer must increase `recovery_kv_lease_ttl`; normal reads must not gain a recovery-state branch just to optimize this rare path.

### Phase 1 tests

At minimum, test:

1. Standby payload round-trip preserves hard pin and soft-pin enabled state without deadline fields.
2. A promoted object with an unremounted memory replica remains unreadable.
3. An expired object lease does not make an unremounted memory replica eligible for batch eviction.
4. Tenant quota eviction and offload-on-evict also skip the unremounted replica.
5. Successful remount makes all associated objects receive a recovery lease.
6. An object with another already-readable memory replica is still refreshed when its segment remounts.
7. Soft-pin enabled objects receive the full soft-pin TTL after remount.
8. Hard-pinned objects remain hard pinned.
9. Failed remount does not publish the segment, clear invalid endpoint state, or grant recovery leases.
10. Multiple matching replicas for one object add one pointer per remount call, not one pointer per replica.
11. A later remount of another segment can extend the same object's lease.
12. Unset recovery TTL falls back to normal read TTL; an explicit value is honored.
13. Benchmark data exists for at least one million and ten million objects, with an optional one hundred million object run on target hardware.

## Phase 2: Runtime Steady Clock

### Runtime types

Introduce local aliases to make clock-domain mistakes visible:

```cpp
using LeaseClock = std::chrono::steady_clock;
using LeaseTimePoint = LeaseClock::time_point;
using LeaseDuration = LeaseClock::duration;
```

The final runtime metadata semantics are:

```text
lease_timeout: LeaseTimePoint
soft_pin_enabled: bool
soft_pin_timeout: optional
hard_pinned: bool
```

`soft_pin_enabled` and `soft_pin_timeout` are separate concepts. Enabled means the object participates in soft-pin policy. The timeout says whether the current soft-pin interval is active. An expired timeout does not permanently disable the object.

### Lease APIs

Keep a TTL-based helper for normal read operations and add a deadline-based helper for batch recovery:

```cpp
void GrantLease(uint64_t lease_ttl_ms,
uint64_t soft_pin_ttl_ms) const;

void GrantLeaseUntil(
LeaseTimePoint lease_deadline,
std::optional soft_pin_deadline) const;
```

`GrantLeaseUntil()` must use max-update semantics. It must never shorten a lease extended concurrently by a read or another recovery operation.

All lease helpers should accept or capture one `steady_clock::now()` per logical operation. Batch eviction samples one `now` and passes it through candidate census and re-validation. Remount computes one deadline before the object loop.

### Migration boundary

The following must migrate:

- `ObjectMetadata::lease_timeout`;
- `ObjectMetadata::soft_pin_timeout`;
- all lease/soft-pin comparison helpers;
- batch-eviction candidate timeout containers and sort keys;
- group lease refresh;
- snapshot cleanup comparisons involving object lease or soft pin.

The following must not be mechanically migrated without a separate audit:

- `put_start_time`;
- replication task start times;
- discarded-replica release deadlines;
- generic task cleanup deadlines;
- unrelated heartbeat, disk, or NoF timers.

The phase must end with a repository-wide audit showing that object lease and soft-pin comparisons no longer mix system and steady clock types.

### Legacy complete-snapshot adapter

Phase 2 must remain independently deployable while the existing complete snapshot wire format still stores system-clock milliseconds.

When writing the legacy format, convert a local steady deadline through one clock pair:

```text
legacy_system_deadline = system_now + (steady_deadline - steady_now)
```

When reading it:

```text
remaining = legacy_system_deadline - restore_system_now
local_steady_deadline = restore_steady_now + remaining
```

This adapter is a compatibility boundary only. Phase 2 must not modify the standby snapshot/oplog wire format or claim that raw steady values are portable between processes.

### Phase 2 tests

Test:

1. Normal `GrantLease()` uses steady time.
2. `GrantLeaseUntil()` never shortens a later deadline.
3. Soft-pin enabled and expired states remain distinct.
4. Runtime lease helpers are unaffected by simulated system-clock jumps.
5. Batch eviction sorts and compares steady deadlines.
6. Group refresh uses one steady `now`.
7. Remount recovery leases use steady deadlines.
8. All Phase 1 recovery and eviction tests continue to pass.
9. Legacy v1 complete snapshots round-trip through the adapter.
10. Expired v1 system deadlines convert to already-expired local steady deadlines.
11. Checked conversion rejects overflow and malformed values.

## Phase 3: Complete MasterService Snapshot Clock Mapping

### Snapshot wire formats

The existing manifest is conceptually:

```text
messagepack|1.0.0|
```

The final format extends it without adding another clock sidecar file:

```text
messagepack|2.0.0|||
```

Version semantics:

- `1.0.0`: metadata lease fields are legacy system-clock milliseconds and no mapping fields exist;
- `2.0.0`: metadata lease fields are steady-clock nanoseconds and the manifest contains the mapping pair;
- unsupported versions, missing fields, malformed integers, and overflow are candidate failures.

The system value is encoded as a signed Unix-epoch nanosecond value on supported Mooncake platforms. The steady value is a signed, normalized nanosecond count from the old process's steady-clock epoch. The raw steady epoch is not meaningful by itself; only the pair is meaningful.

### Snapshot clock context

Each complete snapshot captures one global pair after the snapshot state has been frozen and before metadata encoding:

```text
S0 = snapshot system time
C0 = snapshot steady time
```

The pair is stored in the manifest and is associated with the same frozen metadata payload. It is not sampled per object.

The restore path captures one pair before decoding a candidate:

```text
S1 = restore system time
C1 = restore steady time
```

### Conversion formula

For an object deadline `D0` stored in the old Master's steady domain:

```text
old_system_deadline = S0 + (D0 - C0)
remaining = old_system_deadline - S1
new_steady_deadline = C1 + remaining
```

All arithmetic uses checked signed operations. A deadline whose `remaining` is zero or negative remains expired. It does not receive the default read TTL.

The formula deliberately consumes time from snapshot creation until restore starts. Once the converted deadline is in the new Master's steady domain, metadata decoding, cleanup, and service publication naturally consume additional time.

### Clock trust and skew

Cross-machine conversion assumes that deployment system clocks are reasonably aligned, but the Master does not enforce synchronization or reject a candidate solely because of clock skew:

- if the restoring Master's system clock is ahead, leases appear to expire earlier;
- if it is behind, leases appear to last longer;
- deployment operators choose their clock-synchronization policy;
- aggregate mapping values and unusual remaining durations may be logged for diagnosis, without per-object logging.

If a future deployment requires a stronger guarantee, it will need a trusted external time source or an explicit skew policy. That is outside this RFC.

### Metadata v2 fields

The final time-related metadata fields are:

```text
lease_steady_deadline_ns: int64
soft_pin_enabled: bool
soft_pin_steady_deadline_ns: int64 or explicit absent marker
hard_pinned: bool
```

Existing non-lease metadata retains its meaning. `put_start_time` is not silently changed to a steady value by this RFC.

The decoder must receive the parsed manifest/version and restore clock context. It must not infer whether an integer is system milliseconds or steady nanoseconds from the integer's magnitude or array length alone.

### Snapshot write sequence

```text
freeze snapshot state under snapshot_mutex
-> capture S0/C0 once
-> encode metadata with steady deadlines
-> encode segments and task manager
-> build v2 manifest with S0/C0
-> upload metadata, segments, and task manager
-> publish the manifest last
```

The manifest remains the publication marker. A v2 manifest is not visible until all payloads are available. Backup and child-process paths must preserve the exact manifest contents.

### Snapshot restore and candidate fallback

The repository returns a structured manifest with the downloaded candidate. The decoder receives:

```text
wire_version
optional snapshot clock pair
restore system now
restore steady now
```

Restore tries candidates from newest to oldest. The current candidate fails if:

- the manifest is unsupported or malformed;
- a v2 mapping field is missing or invalid;
- checked clock arithmetic overflows;
- metadata time fields do not match the manifest version;
- metadata or descriptors cannot be reconstructed consistently.

On failure, the current partial state is reset and the next older candidate is attempted. The restore path must not:

- use `now + default_kv_lease_ttl` for a corrupt deadline;
- treat an old raw steady value as the new Master's steady epoch;
- silently parse v2 fields with the v1 decoder;
- skip one corrupted object while claiming a complete snapshot restore.

### Complete snapshot cleanup

Complete snapshot restore does not assign a recovery TTL. It uses converted historical deadlines:

1. Non-hard-pinned metadata whose lease and soft pin are both expired may be cleaned according to existing snapshot cleanup behavior.
2. Hard-pinned metadata remains present when its state is otherwise complete, even if its lease is expired or a memory replica is temporarily unreadable.
3. Hard pin does not hide corrupt payloads, invalid descriptors, or unrecoverable processing state.
4. `soft_pin_enabled` remains true even when its current timeout has expired; it is simply not active until renewed.
5. Standby/oplog unreadable-replica protection is not used to override complete snapshot cleanup. The two paths intentionally have different restore semantics. After a retained object enters normal runtime state, the common ordinary-eviction predicate still applies.

The hard-pin override of lease-expiry cleanup is the chosen initial design and an explicit open review point.

### Phase 3 tests

At minimum, test:

1. v2 manifest encode/decode round-trip.
2. v1 manifest acceptance by the new decoder.
3. Missing clock fields, invalid integers, and checked-arithmetic overflow.
4. Candidate fallback after a malformed mapping.
5. Active lease conversion with a known downtime interval.
6. Restore-decode time continuing to reduce the remaining lease.
7. Expired historical deadlines remaining expired.
8. Active soft-pin conversion and expired soft-pin state.
9. Hard-pinned object retention during snapshot cleanup.
10. Non-hard-pinned expired-object cleanup.
11. v1 system-millisecond snapshots converting to local steady deadlines.
12. Positive and negative system-clock skew following the documented formula.
13. Manifest-last publication and backup preservation.
14. Batch-record oplog mode continuing to avoid the primary complete-snapshot path.

## Compatibility and rollout

### Version matrix

| Deployment state | Standby/oplog recovery | Runtime lease clock | Complete snapshot wire |
| --- | --- | --- | --- |
| Current main + PR #3141 | Transitional recovery protection | `system_clock` | v1 system milliseconds |
| After Phase 1 | Final standby recovery semantics | `system_clock` | v1 system milliseconds |
| After Phase 2 | Final standby recovery semantics | `steady_clock` | v1 through compatibility adapter |
| After Phase 3 | Final standby recovery semantics | `steady_clock` | v2 steady nanoseconds + mapping, v1 read compatibility |

Each phase must start and restore independently. No phase may require a later phase to be deployed before the Master can start, promote, remount, or read an existing supported snapshot.

### Standby/oplog compatibility

The standby reader may assign defaults for fields absent in older payloads:

- missing `hard_pinned` defaults to false;
- missing `soft_pin_enabled` defaults to false;
- missing lease fields are not reconstructed and must not be interpreted as a historical deadline.

The exact standby wire version must be fixed with Phase 1. It must not reuse the complete snapshot's v1/v2 version numbers or clock mapping.

### Complete snapshot compatibility

The new Master must read v1 complete snapshots. A Master that does not understand v2 may reject them. There is no dual-write requirement. Before a rollback to a v1-only binary, operators must retain a usable v1 snapshot or accept the existing empty-state recovery risk.

### PR #3141 integration

The following PR #3141 work is compatible and should remain:

- temporary offset-allocator gap handles are released so restored free capacity is available;
- memory eviction is not armed when the allocation failure cannot be fixed by eviction;
- OpLog reservation failure stops the current eviction cycle and retains the retry trigger;
- submission-result naming and recovery/backpressure comments.

Phase 1 must replace:

- the temporary pre-remount lease that makes an unreadable replica evictable after TTL expiry;
- the `was_readable` transition test and the temporary `unordered_map` collection.

## Performance and resource considerations

### Normal path

There is no new segment-state branch in the normal read path. Normal lease refresh continues to update object metadata, now in the steady-clock domain.

### Remount path

Remount performs:

- the existing metadata scan and descriptor matching;
- one pointer per affected object in a temporary vector;
- one shared recovery-deadline assignment per affected object.

For one hundred million pointers, the vector payload is approximately 800 MB before capacity overhead. This is intentionally less than the multi-gigabyte overhead of a temporary `unordered_map`, but it is still a real recovery peak.

The vector must be released immediately after the assignment. If that peak is not acceptable on target hardware, shard-batched remount publication is a separate design; it must not be introduced implicitly because it changes segment atomicity.

### Observability

Add aggregate metrics, without per-object logs, for:

- remount matched object count;
- recovery-deadline assignment duration;
- total remount duration;
- effective recovery TTL;
- estimated visible TTL at service publication;
- skipped unreadable eviction replicas;
- v1/v2 snapshot restore counts;
- clock-conversion failures;
- snapshot candidate fallbacks.

Warn when deadline assignment consumes a material fraction of the configured recovery TTL or when a snapshot candidate is rejected. Do not log one line per object.

## Concurrency and failure handling

### Remount failure

On allocator, descriptor, or buffer failure:

- do not grant a recovery lease;
- do not clear invalid endpoint state;
- do not publish the segment as serviceable;
- roll back temporary allocator/buffer state through existing RAII and remount cleanup.

### Lease assignment failure

Deadline assignment should not allocate memory or return ordinary business errors. If a parallel worker raises an unexpected exception, treat the remount as failed for publication, emit a partial-assignment warning, and retain the unreadable protection. Exact deadline rollback is unnecessary because an extension cannot cause premature eviction.

### Snapshot mapping failure

An invalid v2 mapping is a candidate-level serialization failure. Reset partial state and try an older candidate. Never replace an invalid historical deadline with a default recovery TTL.

### OpLog backpressure

The PR #3141 ordered-writer reservation/submission result handling remains orthogonal. A reservation failure stops the current eviction cycle and retains the eviction trigger; a submission failure skips the affected object according to the existing policy. The future unreadable-replica predicate must be shared by candidate selection, OpLog descriptor prediction, and final mutation so backpressure handling cannot expose a second eligibility rule.

## Alternatives considered

### Persist lease deadlines in standby snapshot/oplog

Rejected. It creates high-frequency durable churn, makes recovery windows start too early, and preserves a transient cache-retention decision that does not need cross-process durability.

### Assign recovery TTL at the beginning of promotion

Rejected. A large restoration can consume the entire TTL before the segment is serviceable. The deadline starts after successful remount reconstruction.

### Use a segment-level lease or a global recovery epoch

Rejected for this phase. It would require an additional segment/recovery check in common read and lease paths. Independent object deadlines are acceptable because failover is rare and recovery cost is isolated.

### Activate the lease on the first client read

Rejected. It adds read-path state and makes object visibility semantics more complex. The chosen model exposes serviceability through remount and assigns the object lease before publishing the segment.

### Keep `system_clock` for runtime lease comparisons

Rejected. Wall-clock jumps can shorten or extend ordinary leases unpredictably.

### Reset complete snapshot objects to a default recovery TTL

Rejected. It changes historical snapshot semantics and can resurrect expired cache entries.

### Persist raw steady-clock values without a mapping

Rejected. Steady-clock epochs are process-local and cannot be compared directly after failover.

### Add a separate recovery soft-pin TTL

Not chosen. The normal soft-pin TTL is sufficient, and an additional setting would expand configuration without solving a demonstrated problem.

## Open questions

These items are intentionally left open for maintainer review but do not block the three-phase architecture:

1. Should hard pin override complete-snapshot lease-expiry cleanup exactly as specified, or should cleanup retain a narrower exception?
2. Should a future heartbeat carry the set and state of each client segment to clean up live clients whose segments never remount?
3. If a target deployment cannot tolerate the approximately 800 MB pointer-vector peak, should remount gain an explicitly designed shard-batched publication mode?
4. Should group recovery refresh only directly remounted objects, or should a future tested group contract refresh all members atomically?
5. How should disk and NoF recovery define readable and evictable replicas independently of the memory rule?
6. If deployment clock skew becomes operationally unacceptable, should snapshot restore gain a configurable skew warning or an external trusted-time source?

## Acceptance criteria

The RFC is considered implemented when:

1. Standby snapshot/oplog payloads contain no lease or soft-pin deadlines.
2. Hard-pin and soft-pin enabled state survives standby recovery.
3. Unremounted or unreadable memory replicas are excluded from every ordinary memory-eviction path without a TTL.
4. Successful remount unconditionally refreshes every associated object with one batch recovery deadline.
5. The recovery TTL is independently configurable and defaults to normal read TTL.
6. Runtime lease and soft-pin comparisons use `steady_clock` only.
7. Complete snapshot v2 stores steady deadlines and one clock mapping, while v1 snapshots remain readable.
8. Complete snapshot restore consumes downtime and restore time, and never substitutes a recovery TTL for an expired historical deadline.
9. Invalid v2 mapping causes candidate fallback.
10. The three phases are independently buildable, testable, and deployable in the version matrix above.
11. Target-scale remount benchmark data demonstrates that the externally visible recovery window is operationally useful, or the deployment documents a larger recovery TTL.

## Implementation checklist by pull request

### Phase 1: standby snapshot/oplog recovery lease

- [ ] Add `recovery_kv_lease_ttl` configuration and propagation.
- [ ] Add explicit standby `hard_pinned` and `soft_pin_enabled` fields.
- [ ] Remove temporary pre-remount lease protection and its expiry-to-eviction test.
- [ ] Add the unreadable memory-replica guard to every ordinary eviction path.
- [ ] Replace remount `was_readable` logic with a deduplicated object vector.
- [ ] Compute one recovery deadline per remount batch.
- [ ] Unconditionally refresh all objects associated with successfully remounted segments.
- [ ] Preserve full soft-pin TTL for enabled objects.
- [ ] Add failure, rollback, and target-scale benchmark tests.

### Phase 2: runtime steady clock

- [ ] Introduce lease clock aliases and deadline-based grant helper.
- [ ] Migrate object lease and soft-pin fields and all comparison call sites.
- [ ] Keep unrelated time fields in their existing domains until independently audited.
- [ ] Add the v1 complete-snapshot compatibility adapter.
- [ ] Add clock-jump, conversion, and full-regression tests.

### Phase 3: complete MasterService snapshot clock mapping

- [ ] Add manifest v2 parser and structured `SnapshotManifest`.
- [ ] Capture one snapshot system/steady pair under the frozen snapshot context.
- [ ] Encode steady deadlines and explicit soft-pin-enabled state.
- [ ] Convert v2 deadlines into the restoring Master's steady domain.
- [ ] Preserve v1 decoding.
- [ ] Reject malformed mappings and fall back to older candidates.
- [ ] Verify child-process, backup, and manifest-last publication behavior.
- [ ] Add downtime, restore-time, skew, cleanup, and hard-pin tests.

## Conclusion

The proposal puts complexity in the rare recovery paths and keeps normal reads simple:

- standby/oplog recovery does not persist transient deadlines;
- unreadable memory replicas are protected by eviction eligibility, not an expiring temporary lease;
- remount assigns independent object leases after reconstruction succeeds;
- runtime comparisons use a monotonic clock;
- complete MasterService snapshots preserve historical retention through one explicit clock mapping;
- the two snapshot systems remain parallel and independently versioned.

The principal engineering risk is the remount cost for very large segments, especially the temporary object-pointer vector and final deadline assignment. That risk is measurable, observable, and isolated to the recovery path. It must be addressed with benchmark data and, only if necessary, bounded recovery-path parallelism or a deployer-selected recovery TTL, rather than by adding segment checks to normal reads.

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)