cockroachdb / cockroachdb/pebble
db: reduce Ingest I/O with manifest+db locks
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
To identify the target level of M ingested files in a LSM with N levels, we may need to seek in O(N × M) sstables to look for data overlap. This is all performed during application while holding `DB.mu` and the manifest lock, which prevents concurrent flushes and compactions from scheduling or completing while the mutex is held.
Ingest performs these data overlap checks while holding the manifest lock for a consistent view of the LSM. However, for the purposes of detecting data overlap, ingest could be optimistic about its view of the LSM.
When Ingest acquires the manifest lock during the apply step, Ingest can exclude any sstables or memtables with largest sequence numbers already observed during data overlap calculations. Ingest only needs to search in any new sstables for more recent overlap introduced since its optimistic scan.
Related to #25 and [this TODO to reduce overlap checks by using L0 sublevels](https://github.com/cockroachdb/pebble/blob/7b30bd86ff65c3edadda1fea4f1f34f1106193aa/ingest.go#L535).
Also related to #1683 which will remove the strict requirement of preventing file boundary overlap, making target level a function of data overlap exclusively.
Jira issue: PEBBLE-173
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.