cockroachdb / cockroachdb/cockroach
restore: inaccuracy of estimated mvcc stats in online restore confuses kv split queue
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Online restore does not know the exact MVCC stats for the restored data: it knows some detail about the aggregate content chunks of the files it is linking in, but does not and cannot know exactly how many keys in the layers it adds are revisions of other layers vs net new keys, so it cannot perfectly predict the actual mvcc stats. As such, it must estimate. Additionally, since files don't map cleanly to restored spans, it estimated further. In reality, it _really_ estimates since backup's metadata does not map 1:1 with mvcc stats.
Combined, this can mean that online-restored data can have substantial inaccuracy in its mvcc stats.
Generally this was expected/intended: get the data in first, then refine stats over it later. However, if those stats are too incorrect, they have been observed to confuse the split-queue in the interim. One particular insidious behavior appears to be if a range is actually very small -- so small it is one row and has no valid split point -- but the stats say it is very big, it can loop in the split queue due to stats indicating a need to perform an impossible split.
There are likely several aspects to fixing/improving this (which may warrant decomposition into additional sub-issues as we explore them) including a) improving the accuracy of our estimation, b) making the split queue better handle inaccurate stats and c) making the split-queue better handle remote files by reducing the cost of splitting ranges (remote or otherwise) such as by using keys from the LSM manifest or SST index blocks instead of reading -- perhaps from expensive and slow remote storage -- a large number of individual KVs.
Jira issue: CRDB-60221
Epic CRDB-50824
Contributor guide
Assessment
This issue has not been assessed yet.