cockroachdb / cockroachdb/cockroach

backupccl: memory monitored restore processor erroneously restores deleted data

Open
#103,334 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The new memory monitored restored attempts to limit the number of SSTs added to a single iterator. When a given span has more SSTs that would be allowed by the memory monitor, it attempts to process one set of SSTs first, and then the second set.

This, however, poses a problem for deletions. Currently, we do not write deletions directly, rather we depend on the iterator never returning a deleted key. This assumption was correct when all SSTs related to a key were definitely in the same iterator. It is no longer correct when the SSTs for a given iterator can be split over multiple iterators.

**Proposed work to fix this**

To fix this we think we need to (1) ensure that all SSTs for a given _layer_ are all inside the same iterator and (2) change our usage of the iterator to raise deletion tombstones (both point and range keys) and then explicitly write those deletions during the restore process.

- [ ] Unit test that reproduces the between-layer issue
- [ ] Unit test that reproduces in in-layer issue
- [ ] Pass layer information in RestoreSpanSpec
- [ ] Ensure all SSTs from a single layer are added to a restore iterator
- [ ] Raise point deletion tombstones in ReadAsOfInterator
- [ ] Raise range tombstones in ReadAsOfIterator
- [ ] Correctly write range tombstones during restore

See
- https://github.com/cockroachdb/cockroach/pull/93324
- https://github.com/cockroachdb/cockroach/pull/103301

Jira issue: CRDB-27949

Epic CRDB-28050

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.