cockroachdb / cockroachdb/cockroach
storage,kv: investigate overlap memtable during snapshot application
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In the 23.1 test cluster, we're observing higher than expected exercising of the new flushable ingests code path. This code path is triggered one or more of the sstables in an ingest overlaps with one or more of the memtables. In 22.2 and earlier, ingest would flush the memtable, wait for it complete and then proceed. In 23.1 the ingest writes a record committing to the ingest to the WAL and layers the ingested sstables onto the 'flushable' queue containing the memtables and large batches.
The flushable ingest numbers we're observing indicate that many snapshot applications overlap a memtable, forcing memtable rotation and flushes. This is unexpected, because we thought the ingested sstables should have narrow keyspans local to the associated KV range. We expected that the memtable would not contain any keys that fall within the new replica's six keyspans.
Here's a few example version edits pulled from the test cluster.
```
61234027/158636
log-num: 2073703
next-file-num: 2073724
last-seq-num: 15888760204
added: L0 2073696:1535<#15888757362-#15888757362>[/Local/RangeID/191853/r""/0,0#15888757362,RANGEKEYDEL-/Local/RangeID/191853/s""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073701:1180<#15888757363-#15888757363>[/Local/RangeID/191853/u""/0,0#15888757363,RANGEDEL-/Local/RangeID/191853/v""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073697:2387<#15888757364-#15888757364>[/Local/Range/Table/1199/1/2073185522769554992/0,0#15888757364,RANGEKEYDEL-/Local/Range/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073698:1248<#15888757365-#15888757365>[/Local/Lock/Intent/Local/Range/Table/1199/1/2073185522769554992/0,0#15888757365,RANGEKEYDEL-/Local/Lock/Intent/Local/Range/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073699:1224<#15888757366-#15888757366>[/Local/Lock/Intent/Table/1199/1/2073185522769554992/0,0#15888757366,RANGEKEYDEL-/Local/Lock/Intent/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L6 2073700:1200<#15888757367-#15888757367>[/Table/1199/1/2073185522769554992/0,0#15888757367,RANGEKEYDEL-/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
61234765/158637
next-file-num: 2073734
last-seq-num: 15888760945
added: L0 2073717:1540<#15888760939-#15888760939>[/Local/RangeID/191841/r""/0,0#15888760939,RANGEKEYDEL-/Local/RangeID/191841/s""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073722:1179<#15888760940-#15888760940>[/Local/RangeID/191841/u""/0,0#15888760940,RANGEDEL-/Local/RangeID/191841/v""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073718:2938<#15888760941-#15888760941>[/Local/Range/Table/1199/1/2202183733075216192/0,0#15888760941,RANGEKEYDEL-/Local/Range/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073719:1248<#15888760942-#15888760942>[/Local/Lock/Intent/Local/Range/Table/1199/1/2202183733075216192/0,0#15888760942,RANGEKEYDEL-/Local/Lock/Intent/Local/Range/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073720:1224<#15888760943-#15888760943>[/Local/Lock/Intent/Table/1199/1/2202183733075216192/0,0#15888760943,RANGEKEYDEL-/Local/Lock/Intent/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L6 2073721:1200<#15888760944-#15888760944>[/Table/1199/1/2202183733075216192/0,0#15888760944,RANGEKEYDEL-/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
61235499/158638
log-num: 2073723
next-file-num: 2073734
last-seq-num: 15888761334
added: L0 2073725:159667<#15888757368-#15888760198>[/Local/RangeID/2/u/RaftHardState/0,0#15888760084,SET-/Table/13/1/2023-03-22T17:47:10.587605Z/850122606474362886/6/1/1679507230.587605315,0#15888759341,SET] (2023-03-22T17:47:10Z)
61235607/158639
log-num: 2073724
next-file-num: 2073734
last-seq-num: 15888761346
added: L0 2073711:1539<#15888760199-#15888760199>[/Local/RangeID/191857/r""/0,0#15888760199,RANGEKEYDEL-/Local/RangeID/191857/s""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073716:1179<#15888760200-#15888760200>[/Local/RangeID/191857/u""/0,0#15888760200,RANGEDEL-/Local/RangeID/191857/v""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073712:1828<#15888760201-#15888760201>[/Local/Range/Table/1199/1/-3381595941584118608/0,0#15888760201,RANGEKEYDEL-/Local/Range/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073713:1248<#15888760202-#15888760202>[/Local/Lock/Intent/Local/Range/Table/1199/1/-3381595941584118608/0,0#15888760202,RANGEKEYDEL-/Local/Lock/Intent/Local/Range/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073714:1224<#15888760203-#15888760203>[/Local/Lock/Intent/Table/1199/1/-3381595941584118608/0,0#15888760203,RANGEKEYDEL-/Local/Lock/Intent/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L6 2073715:1200<#15888760204-#15888760204>[/Table/1199/1/-3381595941584118608/0,0#15888760204,RANGEKEYDEL-/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
```
There's no specific recording of which version edits correspond to flushable ignests that overlap with the memtable, but I believe all the ones that ratchet `log-num` are, since only flushes should be ratcheting `log-num`.
Jira issue: CRDB-25798
Epic CRDB-27235
Contributor guide
Assessment
This issue has not been assessed yet.