[RFC]: Standby-Generated Snapshots and Bounded OpLog Retention

> Implementation note: discussion after RFC submission changed several delivery details without changing the RFC's primary goal. The implementation keeps the legacy and batch OpLog snapshot systems permanently separate, uses `enable_oplog` as the new-mode switch, writes a 300-million-object state in fixed object-count chunks, and does not remove the legacy reader/writer. The [delivery roadmap](../../../plans/2026-07-22-oplog-ha-pr-delivery-roadmap-zh.md) and PR-N01 through PR-N13 specs are authoritative for implementation order and current progress; the RFC body below remains the submitted design context.

## Summary

This RFC moves HA snapshot generation away from the serving primary and onto standby masters. A standby captures a complete applied batch boundary, writes an immutable snapshot to shared object storage, and publishes it through a fenced etcd transaction. Recovery first tries two independently addressable snapshot pointers and replays the OpLog suffix. If both snapshots are unusable, recovery may fall back to the complete OpLog only when every batch from the initial durable boundary is still present and valid.

Once two snapshots are independently verified, Mooncake may garbage-collect older snapshot objects, publish a monotonic compaction floor, and delete OpLog batches covered by the older protected snapshot. The protocol always publishes the floor before deleting OpLog data. Snapshot GC failures are reported and retried but do not block safe OpLog pruning.

The feature is disabled by default. The first implementation supports etcd only. It does not add a primary fallback writer, an automatic write backpressure controller, or support for Redis-backed snapshot coordination.

## Motivation

The existing primary snapshot path forks the serving process while holding a global snapshot barrier. That design has three long-term problems:

1. Snapshot work competes with serving-primary resources and complicates the primary mutation lock graph.
2. Snapshot descriptors do not carry a complete batch cursor, so a reader cannot unambiguously start suffix replay or prove which batches are safe to remove.
3. OpLog retention is unbounded. Deleting keys without a reader-visible floor would turn legitimate pruning into an indistinguishable recovery gap.

Standbys already maintain replicated object metadata and segment state. They are the natural place to create recovery checkpoints without involving the serving primary.

## Goals

- Generate snapshots only from standby state.
- Capture a state that ends at a complete durable batch boundary.
- Keep the serving primary independent of capture, encoding, and upload work.
- Publish immutable snapshots through a fenced, monotonic etcd protocol.
- Give recovery two independently addressable snapshot candidates.
- Fall back to a complete OpLog when both snapshots are unusable and the full history still exists.
- Bound OpLog and snapshot-object growth without creating a data-loss window.
- Preserve explicit readers for historical snapshot formats during migration.
- Remove the legacy primary snapshot writer and its global snapshot barrier after the new path is rolled out.
- Provide enough metrics, logs, and recovery artifacts to diagnose every stage.

## Non-Goals

- Supporting Redis, K8s objects, or another KV backend as the authoritative snapshot pointer/floor store.
- Making the primary generate snapshots when no standby is healthy.
- Adding an automatic primary write throttler when snapshotting stalls.
- Fixing mutations that are not represented correctly in the durable batch OpLog. This RFC protects the state that the standby has durably applied.
- Introducing client-side snapshot encryption. Transport security, object store encryption, and IAM remain deployment responsibilities.
- Implementing a copy-on-write or MVCC metadata container in the first version.
- Automatically resetting a damaged cluster or accepting a discontinuous OpLog.

## Terminology

- **Durable prefix:** The highest batch and sequence boundary acknowledged as durably persisted by the batch OpLog writer.
- **Applied cursor:** The last complete batch ID and sequence ID successfully applied by a standby.
- **Capture:** An immutable copy of standby state associated with one applied cursor.
- **Candidate snapshot:** An uploaded and locally validated snapshot that is not yet referenced by an authoritative pointer.
- **Latest pointer:** The newest authoritative snapshot reference.
- **Fallback pointer:** The previous independently addressable authoritative snapshot reference.
- **Compaction floor:** The highest batch ID that readers must assume may have been deleted.
- **Protected snapshots:** The valid snapshots referenced by `latest` and `fallback` during a maintenance cycle.
- **Maintenance lease:** The etcd lease that serializes snapshot publication, pointer repair, snapshot GC, and floor advancement.

## Current State and Compatibility Baseline

The existing `StandbySnapshot` contains the applied sequence ID, object metadata, and a segment registry. The existing snapshot catalog can address payloads and descriptors, but its latest marker is not a sufficient HA authority because it has no fenced batch-cursor publication protocol.

PR-N01 introduces an algorithm-agnostic `payload_checksum`, a schema version, `last_included_batch_id`, and `previous_snapshot_id` while preserving the legacy three-field descriptor encoding. Legacy descriptors decode with zero values for all new fields. A schema-v1 descriptor requires a consistent cursor: sequence and batch IDs must be either both zero or both non-zero.

The old primary snapshot reader remains available for explicit import during migration. It is not an automatic authority in pointer mode.

## Design Principles

1. **Pointers, not listing, select recovery state.** Recovery never promotes an object merely because it appears in a bucket or catalog listing.
2. **Immutable objects make partial failure harmless.** An interrupted writer leaves an orphan, not a partially updated authoritative snapshot.
3. **A lease reduces duplicate work; the final transaction provides correctness.** Lease ownership alone is not a publication proof.
4. **Readers learn the floor before data disappears.** The floor is committed before any covered batch is deleted.
5. **Failure retains data.** When validation is inconclusive, Mooncake skips GC and pruning.
6. **The primary is not a snapshot worker.** Lack of a healthy standby causes snapshot stalling and capacity growth, not a primary fallback snapshot.
7. **Recovery does not guess.** Missing or malformed history is never treated as an empty cluster.

## Architecture

```text
Primary batch writer
|
| durable batches + durable prefix
v
etcd <-------------------------------+
| |
| poll/apply | lease + pointer CAS + floor
v |
Standby apply loop -> immutable capture -> snapshot worker
|
| payload/manifest/descriptor
v
shared object store

Recovery:
etcd latest -> exact object GET -> decode -> replay suffix
|
+---- failure -> etcd fallback -> exact object GET -> replay suffix
|
+---- failure -> complete OpLog from batch 1
|
+---- gap -> fail closed
```

The snapshot worker owns only an immutable capture, a temporary encoded file, an object-store client, and its maintenance-lease session. It does not retain a pointer to the standby service or the promoted primary.

## State Contract

### State Persisted in the Snapshot

- Object keys and tenant IDs.
- Object metadata required by the standby and promotion path.
- Replica descriptors and their persisted lifecycle status.
- Segment registry entries visible to the standby.
- Last included sequence ID.
- Last included batch ID.
- Producer view version associated with the captured prefix.
- Payload schema, size, and checksum.

### State Reloaded from an External Authority

- Tenant quota policy from its configured connector.
- Leadership lease and current primary view from the leadership backend.
- Runtime configuration, object-store credentials, and endpoints.

### State Rebuilt or Intentionally Discarded

- Allocator indexes rebuilt from segment and replica descriptors.
- Metrics and derived counters.
- Client liveness and sessions.
- In-flight requests, callbacks, and background tasks.

This RFC guarantees the state represented by the durable batch OpLog and successfully applied by the standby. It does not wait for every later durability track, including stricter Segment lifecycle persist-before-publish work. That limitation must remain visible in release notes and operator documentation.

## Snapshot Descriptor and Payload

### Descriptor

A schema-v1 `SnapshotDescriptor` contains:

```text
snapshot_id
schema_version
last_included_seq
last_included_batch_id
producer_view_version
payload_checksum
payload_size
manifest_key
object_prefix
previous_snapshot_id
created_at_ms
```

PR-N01 establishes the compatible descriptor fields. If `payload_size` is not added to the descriptor itself, it must be present in the versioned pointer and manifest. Readers compare all duplicate identity fields and reject a mismatch.

The checksum string is algorithm-tagged. The exact initial algorithm is chosen by PR-N03 based on SDK/backend support, but it must cover the complete encoded payload and must not use an S3 ETag as a substitute.

### Payload Envelope

The payload uses the existing `StandbySnapshot` reflection with `struct_pack`, wrapped in a small envelope:

```text
magic
envelope_schema_version
payload_codec_version
last_included_seq
last_included_batch_id
uncompressed_size
encoded_size
checksum_algorithm
checksum
encoded StandbySnapshot bytes
```

The envelope duplicates the cursor deliberately. The descriptor, manifest, pointer, and envelope must agree. A mismatch is a hard validation failure.

Unknown envelope versions, truncation, overflow, invalid object counts, or cursor mismatches return a decode error. Readers retain explicit support for the legacy primary snapshot format; writers never emit it.

### Object Layout and Immutability

Each snapshot ID is globally unique within the cluster namespace and includes enough entropy to distinguish concurrent nodes and process incarnations.

```text
///payload
///manifest
///descriptor
```

Existing keys are never overwritten. A key collision fails the candidate. Objects are written in payload, manifest, descriptor order. None becomes authoritative until the etcd pointer transaction succeeds.

## Etcd Keyspace

The existing normalized cluster namespace prefixes all keys. Logical suffixes are:

```text
/snapshot/writer
/snapshot/latest
/snapshot/fallback
/snapshot/compaction_floor
```

`/snapshot/writer` is attached to the maintenance lease and stores a unique, non-reusable owner token. The token is not persisted in the snapshot descriptor.

Both pointer records are independently parseable and contain at least:

```text
pointer_schema_version
snapshot_id
last_included_seq
last_included_batch_id
producer_view_version
payload_size
payload_checksum
created_at_ms
```

The two keys are updated in one etcd transaction. Independent keys allow recovery to locate `fallback` even when the `latest` value itself is malformed or missing.

## Capture Protocol

### Admission

A standby may start a capture only when:

- standby snapshotting is enabled;
- it has applied at least one complete batch, or the S01 zero-state contract proves a legitimate empty cluster;
- its cursor is internally consistent with the applied batch's last sequence;
- its lag at admission is no greater than `max_snapshot_lag_batches`;
- the default maximum lag is 1000 batches;
- memory and temporary-disk budgets permit the operation;
- no local snapshot capture is already running;
- it owns the global maintenance lease.

Lag is measured once at admission as:

```text
durable_prefix.batch_id - standby_applied_batch_id
```

The lag threshold is not rechecked after upload. Rechecking it at publication would starve snapshot creation under sustained writes or a slow object store.

### Complete-Batch Linearization

The capture cursor advances only after the whole batch has applied. A failed or partially applied batch cannot appear in a capture. The implementation must validate that the captured sequence equals the selected batch's `last_seq`.

Capture is valid only if its in-memory copy finishes while the process is still a standby. A local, non-persisted role generation may be read before and after copying; a promotion transition invalidates an overlapping copy without waiting for S3 I/O. No role generation or standby instance token is stored in the descriptor.

Capture must not acquire `MasterService::snapshot_mutex_`. Any standby-local critical section is limited to producing the immutable capture. It must not be held during encoding, checksumming, local-file I/O, S3 I/O, HEAD requests, pointer publication, GC, or retries.

### Promotion During Snapshot Work

If promotion begins before the immutable capture is complete, that capture is discarded. Promotion has priority and must not wait for object-store work.

If the immutable capture completed while the node was still a standby, its worker may finish encoding, upload, and fenced publication after that process has been promoted. This remains a standby-generated snapshot because it never reads promoted-primary state. After capture, the worker must not access or lock the standby service or the new `MasterService`.

The existing `producer_view_version` records the view associated with the captured prefix. No additional `captured_view` field is required.

## Encoding and Resource Bounds

The first implementation accepts that an immutable capture may temporarily duplicate standby metadata. It must avoid a third full in-memory copy:

1. Estimate capture size before copying.
2. Reject the attempt if `max_snapshot_capture_bytes` would be exceeded.
3. Encode the capture to a temporary file while calculating the full-object application checksum.
4. Release the original capture after encoding.
5. Decode and structurally validate the temporary file into a temporary object, then release that object before upload. The capture and decoded copy must not coexist.
6. Stream the temporary file through multipart upload.
7. Remove the file after publication or terminal failure.

The configured work directory must have an explicit free-space check. An OOM or full-disk risk skips the snapshot and emits an alert; it never terminates the serving primary. Copy-on-write/MVCC containers are deferred until measurements show that the approximately two-times standby-metadata peak is unacceptable.

## Upload Verification

Large snapshots are not downloaded in full after every upload when the backend provides a trustworthy full-object checksum.

The required sequence is:

1. Encode, checksum, and decode-check the local temporary file.
2. Upload with an explicit full-object checksum supported by the backend.
3. Let the object store validate the transmitted bytes.
4. Issue a metadata/HEAD request.
5. Compare content length, checksum algorithm, checksum value, and object version ID when available.
6. Upload the small manifest and descriptor and read those small objects back.

S3 ETags are not full-object checksums for multipart uploads or several encryption modes and are never accepted as the payload checksum. The object store interface therefore needs a small verified-upload/HEAD result rather than exposing provider-specific ETag rules.

If the configured S3-compatible or local backend cannot return and validate an expected full-object checksum, Mooncake falls back to a complete payload readback. Failure to perform either verification path prevents publication and prevents the snapshot from participating in pruning.

## Pointer Publication

Publication uses one etcd transaction that compares:

- maintenance lease key value equals the writer's owner token;
- raw `latest` and `fallback` values still equal the values read before upload;
- the candidate cursor is strictly newer than every valid authoritative pointer cursor;
- the candidate descriptor, manifest, and payload metadata agree.

Normal publication performs:

```text
fallback = old latest
latest = new candidate
```

For the first snapshot, `fallback` remains absent. If `latest` is malformed or missing but `fallback` is valid, a verified newer candidate may replace `latest` while preserving the valid fallback. Malformed fallback data is never copied into a new valid slot. The transaction compares the raw observed values so concurrent repair or publication causes a clean CAS failure.

A CAS failure does not retry by overwriting the winner. The candidate remains an orphan for GC. Pointer cursors never regress.

## Scheduling

The standby scheduler is disabled by default and uses no general-purpose scheduler framework. Each node has at most one local attempt. All nodes compete for the shared maintenance lease before expensive capture work.

A normal attempt requires both:

- the configured minimum interval since the shared latest snapshot; and
- the configured minimum batch advancement beyond latest.

A capacity threshold may bypass the time interval when uncompacted batches exceed the configured hard limit, but at least one new complete batch is still required. After lease acquisition, the winner rereads pointers, durable prefix, and trigger conditions.

When no standby is eligible, Mooncake creates no snapshot, advances no floor, and removes no OpLog. The primary never falls back to snapshot generation.

## Recovery Protocol

### Recovery Order

Standby startup uses the following order:

1. Read and parse `latest`.
2. Exact-GET its descriptor, manifest, and payload.
3. Verify ID, schema, size, checksum, cursor agreement, and decode result.
4. If latest fails, independently read and validate `fallback`.
5. If either succeeds, install the state in a temporary recovery context and replay from `snapshot.batch_id + 1`.
6. If both fail, attempt a complete OpLog recovery from the initial batch.
7. If complete OpLog recovery fails, remain non-serving and non-promotable.

Recovery never scans object storage or a catalog for an unreferenced candidate.

### Complete OpLog Fallback

Pure OpLog recovery requires all of the following:

- a valid durable prefix;
- batch IDs continuous from batch 1 through the durable prefix;
- every batch validates its checksum and internal first/last sequence range;
- sequence IDs are continuous within and between batches;
- producer view/fencing data is valid;
- no missing record is interpreted as an empty cluster.

Even when a compaction floor exists, recovery may use the complete OpLog if the actual full history still remains after a floor-before-delete crash. It must prove that history by reading it; it cannot infer completeness from the floor. Legitimate empty-cluster recovery uses the S01 zero-state contract.

### Installation

Snapshot decode and suffix replay occur in a temporary recovery state. The running standby state is replaced only after the candidate and required suffix are valid. Failure does not expose partially restored objects or clear a previously healthy local state.

## Snapshot GC and OpLog Pruning

### Protected Set

Maintenance requires both `latest` and `fallback` to be independently valid and ordered. If either slot is unavailable or invalid, Mooncake performs no floor advancement and no OpLog deletion. Recovery may still use the remaining slot. Later successful publications naturally restore a two-snapshot set.

The newer pointer is `S_new`; the older pointer is `S_safe`. The maximum safe deletion cursor is `S_safe.last_included_batch_id`.

### Snapshot GC

Snapshot GC runs under the same maintenance lease before floor advancement. It protects both pointer targets and deletes only snapshot groups that:

- are not referenced by either authoritative slot;
- are older than `S_safe` or are known orphan candidates;
- are older than the grace period;
- are under the configured, dedicated snapshot prefix.

The default grace period is derived as:

```text
1.5 * configured snapshot persistence interval
```

The multiplier is configurable. This grace protects recent failed uploads and eventually consistent listings. The maintenance lease prevents a completed candidate from racing with GC. Incomplete multipart uploads should additionally use the object store's multipart-abort lifecycle policy.

GC is best effort. Partial delete failure increments metrics and schedules a retry but does not invalidate two already verified protected snapshots.

### Ordered Floor Advancement and Deletion

After GC attempts, Mooncake revalidates the two protected snapshot metadata and executes:

1. CAS `compaction_floor` monotonically to `S_safe.batch_id`.
2. After the floor is visible, delete batch keys through `S_safe.batch_id` with the expected producer-view/fencing condition.

The floor never decreases. If floor publication fails, no OpLog is deleted. If floor publication succeeds and range deletion fails, the result is temporary capacity leakage; retry is safe. Mooncake never deletes batches before publishing the reader-visible floor.

### Lagging Live Standby

A reader whose local cursor is below the floor stops ordinary apply and enters snapshot rebootstrap. It loads a valid authoritative snapshot at or beyond the floor into temporary state and resumes suffix replay. If no valid snapshot is available, it remains non-serving and cannot be promoted.

## Failure Semantics

| Failure | Required behavior |
|---|---|
| No eligible standby | No snapshot, no pruning; alert on growing backlog |
| Admission lag over 1000 by default | Skip the attempt; replication continues |
| Inconsistent batch/sequence cursor | Reject capture; do not move applied cursor |
| Capture memory or disk budget exceeded | Skip and alert; do not risk OOM |
| Promotion overlaps capture copy | Invalidate/yield capture; promotion has priority |
| Promotion after immutable capture | Upload/publish may finish without primary-state access |
| Maintenance lease lost | Final pointer CAS fails; uploaded objects are orphaned |
| Payload upload fails | Pointer unchanged; retry in a later cycle |
| HEAD/checksum mismatch | Pointer unchanged; candidate is unusable |
| Backend lacks checksum metadata | Complete readback is mandatory |
| Pointer CAS loses a race | Do not overwrite winner; orphan candidate |
| Latest snapshot corrupt | Try independent fallback |
| Both snapshots corrupt | Prove complete OpLog recovery or fail closed |
| Snapshot GC partially fails | Alert/retry; safe OpLog pruning may continue |
| Floor CAS fails | Delete no OpLog |
| OpLog range delete fails after floor | Retain extra data and retry |
| Standby cursor below floor | Rebootstrap; never skip directly to floor |
| etcd approaches quota | Metrics/alerts only; existing writer fail-stop handles persistence failure |

## Configuration

Names are illustrative until the implementation follows the repository's existing configuration conventions.

| Setting | Default | Meaning |
|---|---:|---|
| `enable_standby_snapshot` | `false` | Enables standby scheduling and publication |
| `snapshot_interval_seconds` | existing configured interval | Minimum normal persistence interval |
| `snapshot_min_advance_batches` | deployment value | Minimum normal cursor advancement |
| `max_snapshot_lag_batches` | `1000` | Admission-only maximum lag |
| `max_uncompacted_batches` | deployment value | Capacity trigger that may bypass interval |
| `max_snapshot_capture_bytes` | deployment value | Hard capture memory budget |
| `snapshot_work_dir` | unset | Temporary encoding directory; required in production |
| `snapshot_gc_grace_multiplier` | `1.5` | Grace relative to snapshot interval |
| `enable_snapshot_gc` | `false` | Enables old/orphan snapshot deletion |
| `enable_oplog_pruning` | `false` | Enables floor advancement and range deletion |
| `snapshot_bootstrap_mode` | `pointer` when enabled | `pointer` or explicit legacy import mode |

Invalid combinations fail at startup. Enabling pointer publication, GC, or pruning without etcd and a shared object store is an error. Production pointer mode must not default to a node-local directory.

## Shared Object Store Requirements

- All current and future master nodes can address the same bucket and prefix.
- Exact-key PUT, GET, HEAD, and DELETE are available.
- Successful PUT followed by exact-key GET/HEAD observes the object.
- Payload keys are immutable and unique.
- Server-side full-object checksum validation is preferred.
- LIST may be eventually consistent because it is used only for conservative GC under a lease and grace period.
- The bucket/prefix is dedicated to Mooncake snapshots.
- Endpoint, bucket, prefix, or credential migration preserves access to both authoritative snapshots until new snapshots replace them.

Local files are supported for tests, single-node development, or an explicitly shared filesystem. A node-local directory is not a supported production HA store.

## Security and Access Control

- Use TLS for etcd and object-store traffic in production.
- Grant the snapshot writer only the required bucket/prefix operations.
- Grant recovery nodes exact-key read access to the protected snapshots.
- Do not log credentials, signed URLs, payload content, or tenant keys.
- Use bucket encryption/SSE according to deployment policy.
- Treat snapshot payloads as sensitive metadata even though they do not carry user object bytes.
- Restrict destructive namespace reset and GC operations to an operator role.

## Observability

At minimum expose:

- capture attempts, skips, successes, and failures by reason;
- capture copy time, encode time, upload time, HEAD verification time, and pointer-CAS time;
- captured bytes, temporary-file bytes, and peak estimated capture memory;
- standby lag at admission;
- latest/fallback IDs, cursors, ages, and validation status;
- last successful snapshot and last failed phase;
- maintenance lease acquisition and loss;
- pointer CAS conflicts and orphan candidates;
- bootstrap source (`latest`, `fallback`, or full OpLog);
- bootstrap download, decode, and suffix replay duration;
- compaction floor and uncompacted batch count;
- snapshot GC object/byte counts and failures;
- OpLog range-delete counts and failures;
- etcd `dbSize`, `dbSizeInUse`, quota, revision, and alarms in capacity tests;
- `snapshot_stalled` and an estimated capacity exhaustion horizon when practical.

Logs identify snapshot ID, cursor, producer view, phase, and stable error code. They do not log payloads or secrets.

## Capacity and Backpressure

This RFC adds no automatic primary backpressure controller. When snapshots stall, the primary continues to write while metrics and alerts show increasing uncompacted history. Existing durable-writer fail-stop behavior remains the last correctness boundary if etcd persistence fails.

Operators may restore the object store, add/repair a standby, increase etcd capacity, manually limit traffic, or perform an explicitly destructive reset. The operational guide covers etcd MVCC compaction, per-member defragmentation, quota alarms, and NOSPACE recovery rather than adding another maintenance controller to Mooncake.

## Upgrade, Rollback, and Destructive Recovery

### Recommended Non-Destructive Rollout

A conservative rollout deploys readers first, enables standby publication without pruning, produces two snapshots, performs a cold-bootstrap drill, and only then enables GC and pruning. This is a recommendation, not a mandatory runtime orchestration protocol.

After the first floor advancement and successful OpLog deletion, rollback to a binary that cannot read the new pointer/floor protocol is unsupported. Rollback must use a version that retains the new reader.

### Destructive Recovery

Deployments that accept data loss may reset instead of preserving rollback compatibility. The recommended method is a new cluster namespace.

In-place reset is permitted only when all writers and standbys are stopped and fenced from reconnecting. The operator must remove snapshot pointers, floor, leases, OpLog keys, and snapshot objects, then run linearizable emptiness checks over every managed prefix. Any residue causes startup to fail and requires the operator to continue cleanup. Stopping processes, revoking leases, and when necessary rotating credentials are required to prevent a stale process from rewriting the old namespace after the check.

Mooncake never initiates destructive recovery automatically.

## Legacy Snapshot Removal and Lock Cleanup

The legacy snapshot-only mode (`snapshot enabled`, `OpLog disabled`) is deprecated by this RFC. The proposed delivery does not retain a formal release window; this remains an explicit open question for maintainers.

After pointer mode, GC, pruning, and catalog-fallback cleanup are complete:

- reject the legacy snapshot-only configuration at startup;
- remove the primary fork snapshot manager and primary snapshot scheduler;
- keep legacy payload/descriptor readers for explicit import;
- audit the two current unique-lock users of `MasterService::snapshot_mutex_`;
- move any real client/segment concurrency requirement to domain-specific locks;
- remove the global snapshot mutex and the many primary mutation shared-locks;
- add concurrent mutation, remount/unmount, promotion, and stress coverage.

Standby capture must never depend on the primary snapshot mutex, so this cleanup does not change the new snapshot protocol.

## Testing Strategy

### Unit Tests

- Legacy and schema-v1 descriptor round trips.
- Empty and non-empty cursor combinations.
- Unknown versions, malformed fields, overflow, truncation, and checksum shape.
- Payload envelope round trip and cross-field cursor mismatch.
- Complete-batch capture and partial-batch rejection.
- Lag admission at 999, 1000, and 1001 batches.
- Local role transition during capture.
- Memory/disk budget rejection.
- Verified upload, HEAD mismatch, and complete-readback fallback.
- Dual-pointer first publish, normal rotation, malformed latest repair, CAS conflict, and non-regression.
- Latest/fallback recovery and complete-OpLog fallback.
- Floor boundary and exact range-delete key limits.
- GC protected-set, grace-period, orphan, partial failure, and retry behavior.

### Concurrency and Failure Injection

- Two standbys competing for the maintenance lease.
- Lease expiry during capture, multipart upload, HEAD, and final CAS.
- Promotion before capture completion and after immutable capture.
- Crash after each object write and before/after pointer CAS.
- Crash after GC and before floor CAS.
- Crash after floor CAS and before OpLog deletion.
- Concurrent new batch append during range deletion.
- Latest/fallback corruption independently and together.
- Reader crossing the floor while live.

### Integration and E2E

- Real etcd, shared S3-compatible storage, process restart, suffix replay, and promotion.
- High write rate with snapshot admission lag near the threshold.
- Large snapshot memory/disk/network measurements.
- Old reader fixtures and explicit legacy import.
- Snapshot-only configuration rejection after legacy removal.
- Capacity soak covering key count, MVCC compaction, defrag, quota, and alarms.
- A saved artifact bundle containing pointers, checksums, cursors, floor, timings, and final acknowledged-data audit.

## Revised Delivery Sequence

This RFC revises the N-series order to include object GC and legacy lock cleanup. After RFC approval, the independent PR specs and roadmap must be realigned to this sequence before N02 development starts:

1. **N01 - Descriptor and cursor compatibility.** Schema-v1 fields, compatible codec, validation, and backend round trips. Implemented on `dev/oplog-ha-prs/N01`.
2. **N02 - Complete batch-boundary capture.** Immutable capture and cursor consistency without primary snapshot locks.
3. **N03 - Streaming codec and verified immutable object writer.** Temporary file, application checksum, server checksum/HEAD, and readback fallback.
4. **N04 - Maintenance lease and dual pointer CAS.** Independent latest and fallback keys with fencing and repair rules.
5. **N05 - Pointer bootstrap and suffix replay.** Latest, fallback, then proven complete OpLog.
6. **N06 - Standby scheduler and resource gates.** Default-off scheduler, admission lag default 1000, time/advance/capacity triggers.
7. **N07 - Real-etcd cold-bootstrap E2E.** Process crash, restart, replay, and promotion artifacts.
8. **N08 - Floor-aware live rebootstrap.** Lagging readers replace state atomically or fail closed.
9. **N09 - Guarded batch range-delete primitive.** Exact upper bound and producer-view/fencing checks; no orchestration.
10. **N10 - Snapshot object GC.** Protected set, 1.5x interval grace, orphan cleanup, and best-effort failure semantics.
11. **N11 - Two-snapshot floor and ordered pruning.** Revalidate, run GC, publish floor, then call guarded range delete.
12. **N12 - Etcd capacity operations.** MVCC compaction, defrag, quota, NOSPACE, and soak documentation/tests.
13. **N13 - Remove automatic catalog fallback.** Pointer-only production authority while retaining explicit legacy import.
14. **N14 - Remove the legacy primary snapshot writer and global barrier.** Reject snapshot-only configuration, replace accidental domain locking, and delete `snapshot_mutex_`.

PRs that do not share production files may still be developed in parallel, but publication, reader rollout, floor advancement, and cleanup preserve the above dependency order.

## Alternatives Considered

### Keep the Primary as a Fallback Snapshot Writer

Rejected. It reintroduces primary resource contention and two authoritative capture implementations. With no eligible standby, Mooncake retains OpLog and alerts instead.

### Prevent a Snapshotting Standby from Participating in Election

Rejected. A slow or stuck object store would delay failover, especially in a single-standby deployment. Only the in-memory capture must complete while the node is a standby; persistence may finish after promotion without touching primary state.

### Use One Pointer with a Predecessor Chain

Rejected. Corruption of the latest pointer value can make the predecessor ID unlocatable. Two independent etcd slots provide a bounded recovery choice.

### Scan the Catalog or Bucket During Recovery

Rejected. Listing cannot establish authority and may surface partial or orphan objects. Listing is only a conservative GC input.

### Full Payload Readback After Every Upload

Rejected as the default because it doubles large-snapshot network I/O. Explicit server-side checksum validation plus HEAD metadata is sufficient; full readback remains the compatibility fallback.

### Use S3 ETag as the Payload Checksum

Rejected. Multipart and encryption modes do not provide a full-object MD5 through ETag.

### Require Complete Standby Catch-Up

Rejected. A snapshot lagging by a bounded number of batches is still a valid prefix. The default admission limit is 1000 batches and suffix replay covers the difference.

### Add Automatic Write Backpressure

Rejected from this series. Metrics, alerts, operator action, and existing writer fail-stop behavior keep the snapshot design smaller.

### Support Every HA Backend in the First Version

Rejected. The protocol depends on etcd transaction, lease, range-delete, MVCC, and maintenance semantics. Other backends require separate designs.

## User and Operator FAQ

### Does the primary ever write a new pointer snapshot?

No. Only a state captured completely while a node was a standby is eligible.

### Must the standby be fully caught up?

No. It must be within `max_snapshot_lag_batches` at admission, default 1000.

### Does promotion wait for S3 upload?

No. A completed immutable capture may continue independently. An overlapping incomplete capture is invalidated so promotion takes priority.

### Why keep two pointers?

They provide an independent fallback and allow pruning only when two protected snapshots exist.

### What happens if both snapshots are corrupt?

Mooncake proves whether the complete OpLog remains. If it does not, the standby stays non-serving and non-promotable until repaired or destructively reset.

### Is every snapshot downloaded after upload?

Not when the backend validates a full-object checksum and exposes it through HEAD. Backends without that capability require a complete readback.

### Does snapshot GC failure stop OpLog cleanup?

No, provided latest and fallback were independently revalidated. GC failure is a capacity leak, not a reason to retain already covered OpLog indefinitely.

### Can a local directory be used?

Only for tests, development, or a deliberately shared filesystem. A node-local directory is not a production HA snapshot store.

### What happens when no standby can snapshot?

No new floor is published and no OpLog is deleted. Capacity metrics and alerts increase; the primary does not take over snapshot generation.

### Can an old binary be restored after pruning starts?

Only if it understands the new pointer, payload, and floor protocol. Operators who accept data loss may reset into a clean namespace.

### Does this snapshot every acknowledged runtime detail?

It snapshots the durable state represented and applied by the batch OpLog. Other durability tracks remain separate.

## Developer FAQ and Invariants

### What is the capture linearization point?

The successful validation of an immutable copy against one complete applied batch cursor while the local role is still standby.

### What makes publication authoritative?

Only the etcd transaction that compares lease ownership and both observed pointer values, then writes the new latest/fallback state.

### May a writer overwrite an existing snapshot key?

No. Snapshot IDs and object keys are immutable.

### May recovery use an orphan if its checksum is valid?

No. Integrity does not establish authority.

### May a reader skip directly to the floor?

No. It must install a snapshot covering the floor or prove a complete OpLog.

### What is the safe deletion boundary?

The older of two valid protected snapshots, inclusive by batch ID. Suffix replay starts at the next batch.

### Why does GC precede floor publication?

It reduces object growth while the full OpLog still exists. A crash after GC but before the floor leaves complete history; GC failure does not block safe floor advancement.

### Which code may acquire the legacy primary snapshot mutex?

No new N-series code. N14 removes that mutex after domain-lock audit.

## Open Questions

1. **Immediate legacy removal:** Should N14 immediately reject and remove the legacy snapshot-only writer after N13, or should maintainers require a deprecation release despite the current proposal for no window?
2. **Checksum algorithm:** Which full-object algorithm gives the best portable behavior across the supported AWS SDK and expected S3-compatible stores without requiring a second payload pass?
3. **Default resource budgets:** What capture-memory limit, temporary-disk reserve, minimum batch advancement, and uncompacted-batch trigger fit common production deployments?
4. **Large-state capture:** Do scale tests confirm that the accepted approximately two-times metadata memory peak is practical, or must a later COW/MVCC capture design become a production prerequisite?
5. **In-place destructive reset:** Should production deployments require credential rotation in addition to process shutdown, lease revocation, and linearizable empty-prefix checks?

## References

- [Amazon S3: Checking object integrity for data uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html)
- [etcd v3.7 API: transactions and leases](https://etcd.io/docs/v3.7/learning/api/)
- [etcd v3.7 maintenance guidance](https://etcd.io/docs/v3.7/op-guide/maintenance/)

## Acceptance Criteria

The RFC is implemented when:

- only standby-captured immutable state can produce pointer snapshots;
- latest and fallback are independently recoverable and atomically rotated;
- upload verification avoids full readback only when an explicit backend checksum proves the full object;
- recovery follows latest, fallback, complete OpLog, then fail closed;
- two protected snapshots exist before any floor advancement;
- snapshot GC precedes pruning and cannot delete protected objects;
- the floor is visible before covered batches disappear;
- a lagging standby can rebootstrap without accepting a gap;
- default-off configuration preserves existing behavior until explicitly enabled;
- real-etcd E2E evidence covers publication, restart, suffix replay, promotion, crash windows, and capacity behavior;
- the legacy primary writer and global snapshot barrier are removed through the final cleanup PR while historical readers remain available.