cockroachdb / cockroachdb/cockroach
kvserver/loqrecovery: support separated engines
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary:**
The loss-of-quorum recovery tooling reads/writes all replica state through a single engine handle. With separated engines (sep-raft-log), plan application writes raft-owned keys (`HardState`, `RaftReplicaID`, recovery evidence records) to the state engine while their readers use the log engine, so a staged plan "applies" without actually recovering the replicas.
Enabling separated engines in `TestHalfOnlineLossOfQuorumRecoveryOnDisk` makes the surviving node hang on restart.
**Findings:**
- Plan application (`MaybeApplyPendingRecoveryPlan` in [loqrecovery/apply.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/loqrecovery/apply.go), and offline `applyRecoveryToLocalStore` in [cli/debug_recover_loss_of_quorum.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/cli/debug_recover_loss_of_quorum.go)) uses one `TODOBothEngines()` batch per store, mixing state-engine keys (descriptor, intent resolution, MVCC stats) with log-engine keys.
- Online collection (`ServeLocalReplicas` in [loqrecovery/server.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/loqrecovery/server.go)) uses a single `TODOBothEngines()` snapshot as both state and raft reader; needs a consistent snapshot pair.
- The offline `debug recover` commands open the store directory as a single engine, so they never construct a separated `Engines` pair (the collection itself is already engine-aware).
- Committing two batches is not atomic: needs a defined commit order and idempotent re-application.
**Next Steps:**
- [ ] Route keys in `applyReplicaUpdate` to separate state/log batches, committed in a crash-safe order (online and offline paths)
- [ ] Open both engines in the offline `debug recover` commands
- [ ] Use a consistent snapshot pair in `ServeLocalReplicas`
- [ ] Re-enable separated engines in the LOQ recovery tests
Epic: CRDB-66033
Jira issue: CRDB-66936
Contributor guide
Research direction
Read loqrecovery/apply.go, pkg/cli/debug_recover_loss_of_quorum.go, and loqrecovery/server.go, starting with MaybeApplyPendingRecoveryPlan, applyRecoveryToLocalStore, and ServeLocalReplicas. Run TestHalfOnlineLossOfQuorumRecoveryOnDisk with separated engines enabled. Done means online and offline recovery handle separated engines, snapshots are consistent, re-application is safe, and the recovery tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100