cockroachdb / cockroachdb/cockroach

storage,kv: investigate overlap memtable during snapshot application

Open
#99,273 24 comments 0 reactions 0 assignees View on GitHub
A-kv-replication A-storage C-investigation O-testcluster T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In the 23.1 test cluster, we're observing higher than expected exercising of the new flushable ingests code path. This code path is triggered one or more of the sstables in an ingest overlaps with one or more of the memtables. In 22.2 and earlier, ingest would flush the memtable, wait for it complete and then proceed. In 23.1 the ingest writes a record committing to the ingest to the WAL and layers the ingested sstables onto the 'flushable' queue containing the memtables and large batches.

The flushable ingest numbers we're observing indicate that many snapshot applications overlap a memtable, forcing memtable rotation and flushes. This is unexpected, because we thought the ingested sstables should have narrow keyspans local to the associated KV range. We expected that the memtable would not contain any keys that fall within the new replica's six keyspans.

Here's a few example version edits pulled from the test cluster.

```
61234027/158636
log-num: 2073703
next-file-num: 2073724
last-seq-num: 15888760204
added: L0 2073696:1535<#15888757362-#15888757362>[/Local/RangeID/191853/r""/0,0#15888757362,RANGEKEYDEL-/Local/RangeID/191853/s""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073701:1180<#15888757363-#15888757363>[/Local/RangeID/191853/u""/0,0#15888757363,RANGEDEL-/Local/RangeID/191853/v""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073697:2387<#15888757364-#15888757364>[/Local/Range/Table/1199/1/2073185522769554992/0,0#15888757364,RANGEKEYDEL-/Local/Range/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073698:1248<#15888757365-#15888757365>[/Local/Lock/Intent/Local/Range/Table/1199/1/2073185522769554992/0,0#15888757365,RANGEKEYDEL-/Local/Lock/Intent/Local/Range/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073699:1224<#15888757366-#15888757366>[/Local/Lock/Intent/Table/1199/1/2073185522769554992/0,0#15888757366,RANGEKEYDEL-/Local/Lock/Intent/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L6 2073700:1200<#15888757367-#15888757367>[/Table/1199/1/2073185522769554992/0,0#15888757367,RANGEKEYDEL-/Table/1199/1/2110042154285458192/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)

61234765/158637
next-file-num: 2073734
last-seq-num: 15888760945
added: L0 2073717:1540<#15888760939-#15888760939>[/Local/RangeID/191841/r""/0,0#15888760939,RANGEKEYDEL-/Local/RangeID/191841/s""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073722:1179<#15888760940-#15888760940>[/Local/RangeID/191841/u""/0,0#15888760940,RANGEDEL-/Local/RangeID/191841/v""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073718:2938<#15888760941-#15888760941>[/Local/Range/Table/1199/1/2202183733075216192/0,0#15888760941,RANGEKEYDEL-/Local/Range/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073719:1248<#15888760942-#15888760942>[/Local/Lock/Intent/Local/Range/Table/1199/1/2202183733075216192/0,0#15888760942,RANGEKEYDEL-/Local/Lock/Intent/Local/Range/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073720:1224<#15888760943-#15888760943>[/Local/Lock/Intent/Table/1199/1/2202183733075216192/0,0#15888760943,RANGEKEYDEL-/Local/Lock/Intent/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L6 2073721:1200<#15888760944-#15888760944>[/Table/1199/1/2202183733075216192/0,0#15888760944,RANGEKEYDEL-/Table/1199/1/2220612048833167792/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)

61235499/158638
log-num: 2073723
next-file-num: 2073734
last-seq-num: 15888761334
added: L0 2073725:159667<#15888757368-#15888760198>[/Local/RangeID/2/u/RaftHardState/0,0#15888760084,SET-/Table/13/1/2023-03-22T17:47:10.587605Z/850122606474362886/6/1/1679507230.587605315,0#15888759341,SET] (2023-03-22T17:47:10Z)

61235607/158639
log-num: 2073724
next-file-num: 2073734
last-seq-num: 15888761346
added: L0 2073711:1539<#15888760199-#15888760199>[/Local/RangeID/191857/r""/0,0#15888760199,RANGEKEYDEL-/Local/RangeID/191857/s""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073716:1179<#15888760200-#15888760200>[/Local/RangeID/191857/u""/0,0#15888760200,RANGEDEL-/Local/RangeID/191857/v""/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073712:1828<#15888760201-#15888760201>[/Local/Range/Table/1199/1/-3381595941584118608/0,0#15888760201,RANGEKEYDEL-/Local/Range/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073713:1248<#15888760202-#15888760202>[/Local/Lock/Intent/Local/Range/Table/1199/1/-3381595941584118608/0,0#15888760202,RANGEKEYDEL-/Local/Lock/Intent/Local/Range/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L0 2073714:1224<#15888760203-#15888760203>[/Local/Lock/Intent/Table/1199/1/-3381595941584118608/0,0#15888760203,RANGEKEYDEL-/Local/Lock/Intent/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
added: L6 2073715:1200<#15888760204-#15888760204>[/Table/1199/1/-3381595941584118608/0,0#15888760204,RANGEKEYDEL-/Table/1199/1/-3363167625826167008/0,0#inf,RANGEDEL] (2023-03-22T17:47:10Z)
```

There's no specific recording of which version edits correspond to flushable ignests that overlap with the memtable, but I believe all the ones that ratchet `log-num` are, since only flushes should be ratcheting `log-num`.

Jira issue: CRDB-25798

Epic CRDB-27235

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.