cockroachdb / cockroachdb/cockroach

kvserver/loqrecovery: support separated engines

Open
#173,627 0 comments 0 reactions 0 assignees View on GitHub
A-kv-recovery A-kv-replication C-enhancement O-agent T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary:**

The loss-of-quorum recovery tooling reads/writes all replica state through a single engine handle. With separated engines (sep-raft-log), plan application writes raft-owned keys (`HardState`, `RaftReplicaID`, recovery evidence records) to the state engine while their readers use the log engine, so a staged plan "applies" without actually recovering the replicas.

Enabling separated engines in `TestHalfOnlineLossOfQuorumRecoveryOnDisk` makes the surviving node hang on restart.

**Findings:**

- Plan application (`MaybeApplyPendingRecoveryPlan` in [loqrecovery/apply.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/loqrecovery/apply.go), and offline `applyRecoveryToLocalStore` in [cli/debug_recover_loss_of_quorum.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/cli/debug_recover_loss_of_quorum.go)) uses one `TODOBothEngines()` batch per store, mixing state-engine keys (descriptor, intent resolution, MVCC stats) with log-engine keys.
- Online collection (`ServeLocalReplicas` in [loqrecovery/server.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/loqrecovery/server.go)) uses a single `TODOBothEngines()` snapshot as both state and raft reader; needs a consistent snapshot pair.
- The offline `debug recover` commands open the store directory as a single engine, so they never construct a separated `Engines` pair (the collection itself is already engine-aware).
- Committing two batches is not atomic: needs a defined commit order and idempotent re-application.

**Next Steps:**

- [ ] Route keys in `applyReplicaUpdate` to separate state/log batches, committed in a crash-safe order (online and offline paths)
- [ ] Open both engines in the offline `debug recover` commands
- [ ] Use a consistent snapshot pair in `ServeLocalReplicas`
- [ ] Re-enable separated engines in the LOQ recovery tests

Epic: CRDB-66033

Jira issue: CRDB-66936

Contributor guide

Open the contributing guide

Research direction

Read loqrecovery/apply.go, pkg/cli/debug_recover_loss_of_quorum.go, and loqrecovery/server.go, starting with MaybeApplyPendingRecoveryPlan, applyRecoveryToLocalStore, and ServeLocalReplicas. Run TestHalfOnlineLossOfQuorumRecoveryOnDisk with separated engines enabled. Done means online and offline recovery handle separated engines, snapshots are consistent, re-application is safe, and the recovery tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.