cockroachdb / cockroachdb/pebble
smart filter block loading
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
This is inspired from https://github.com/cockroachlabs/support/issues/2992 and https://github.com/cockroachdb/pebble/issues/3787
We can do a much better job at loading filter blocks. We currently either use them (blocking a Get) or don't use them at all. Ideally, we would load larger blocks that are valuable in the background.
Here's a proposal:
- blocks not in L6 and under a certain size (e.g. 128KB) are loaded inline with the query, like today
- larger blocks we do not load immediately but maintain a metadata-only cache for them, where we record how many times they would have been useful - i.e. how many `SeekPrefixGE` found no results with the prefix. Once we have enough "hits", we load the block in the background. The number of hits required should be proportional to the size (e.g. must have `size/128KB` recent "hits").
- ideally for the block cache, only the times where the block actually helped exclude something should be counted as a "hit"
https://github.com/cockroachdb/pebble/issues/3734 should also be kept in mind. We might eventually have multiple filter blocks of different sizes which we use in various combinations depending on level.
Jira issue: PEBBLE-226
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.