cockroachdb / cockroachdb/cockroach
backupccl: memory monitored restore processor erroneously restores deleted data
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The new memory monitored restored attempts to limit the number of SSTs added to a single iterator. When a given span has more SSTs that would be allowed by the memory monitor, it attempts to process one set of SSTs first, and then the second set.
This, however, poses a problem for deletions. Currently, we do not write deletions directly, rather we depend on the iterator never returning a deleted key. This assumption was correct when all SSTs related to a key were definitely in the same iterator. It is no longer correct when the SSTs for a given iterator can be split over multiple iterators.
**Proposed work to fix this**
To fix this we think we need to (1) ensure that all SSTs for a given _layer_ are all inside the same iterator and (2) change our usage of the iterator to raise deletion tombstones (both point and range keys) and then explicitly write those deletions during the restore process.
- [ ] Unit test that reproduces the between-layer issue
- [ ] Unit test that reproduces in in-layer issue
- [ ] Pass layer information in RestoreSpanSpec
- [ ] Ensure all SSTs from a single layer are added to a restore iterator
- [ ] Raise point deletion tombstones in ReadAsOfInterator
- [ ] Raise range tombstones in ReadAsOfIterator
- [ ] Correctly write range tombstones during restore
See
- https://github.com/cockroachdb/cockroach/pull/93324
- https://github.com/cockroachdb/cockroach/pull/103301
Jira issue: CRDB-27949
Epic CRDB-28050
Contributor guide
Assessment
This issue has not been assessed yet.