cockroachdb / cockroachdb/cockroach

restore: inaccuracy of estimated mvcc stats in online restore confuses kv split queue

Open
#163,444 1 comment 0 reactions 1 assignee Claimed by @dt View on GitHub
A-disaster-recovery C-enhancement P-2 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Online restore does not know the exact MVCC stats for the restored data: it knows some detail about the aggregate content chunks of the files it is linking in, but does not and cannot know exactly how many keys in the layers it adds are revisions of other layers vs net new keys, so it cannot perfectly predict the actual mvcc stats. As such, it must estimate. Additionally, since files don't map cleanly to restored spans, it estimated further. In reality, it _really_ estimates since backup's metadata does not map 1:1 with mvcc stats.

Combined, this can mean that online-restored data can have substantial inaccuracy in its mvcc stats.

Generally this was expected/intended: get the data in first, then refine stats over it later. However, if those stats are too incorrect, they have been observed to confuse the split-queue in the interim. One particular insidious behavior appears to be if a range is actually very small -- so small it is one row and has no valid split point -- but the stats say it is very big, it can loop in the split queue due to stats indicating a need to perform an impossible split.

There are likely several aspects to fixing/improving this (which may warrant decomposition into additional sub-issues as we explore them) including a) improving the accuracy of our estimation, b) making the split queue better handle inaccurate stats and c) making the split-queue better handle remote files by reducing the cost of splitting ranges (remote or otherwise) such as by using keys from the LSM manifest or SST index blocks instead of reading -- perhaps from expensive and slow remote storage -- a large number of individual KVs.

Jira issue: CRDB-60221

Epic CRDB-50824

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.