cockroachdb / cockroachdb/cockroach

Online Restore Recovery: Torn Transaction Records

Open
#144,408 6 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery A-kv branch-master C-bug P-2 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

As of [PR #144244](https://github.com/cockroachdb/cockroach/pull/144244), online restore supports deleting restoring tables that reference corrupt or unavailable remote SSTs.

In version 25.2, only "offline" online restore will be supported—descriptors remain offline until the download completes. Full online restore will be supported in the future, but it introduces complications with the recovery protocol.

Currently, [a transaction record is stored in the first range written by the transaction](https://www.cockroachlabs.com/docs/v24.3/architecture/transaction-layer#writing). If that range becomes unavailable and is excised, its transaction record is deleted. This is typically safe because the associated data is also removed. However, problems arise if the first write is to a table restored online, and later writes target tables outside the restore.

Example:

t0: transaction record, row, and intent written to range A
t1: row and intent written to range B
t2: row and intent written to range C
t3: transaction is committed
t4: intent on range B is resolved
t5: range A is excised

In this case, the row in range B is committed, but the data in range C is rolled back.

A similar issue occurs with survive zone and survive region tables. A transaction recorded on a survive zone range can cover writes to a survive region range. Both issues could be addressed simultaneously: if a transaction spans ranges with varying durability (e.g., remote survive zone < survive zone < remote survive region < survive region), the transaction record should be stored in the most durable range.

Jira issue: CRDB-49370

Epic CRDB-50824

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.