cockroachdb / cockroachdb/pebble
implement a bloom filter cache for L6
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
We want to make use of L6 bloom filters, but these blocks can be big (much bigger than data blocks) and for large LSMs are unlikely to fit in the block cache. Using them without care could cause significant degradation of random workloads.
We want to find blocks that are actually accessed frequently enough to make up for the overhead of loading them. To achieve this we can implement a special cache that tracks filter block cache misses for L6 (or L5 and L6).
~The specific proposal is to use a byte-budgeted FIFO queue (for each block cache shard). The queue contains just filter block references, and is sized so that the total block sizes in the queue is below e.g. 20% of the block cache size. Each entry has a counter that is incremented every time we tried to access that block. Once the count reaches a threshold (e.g. ` / `), we asynchronously load that block into the block cache. The ghost cache approximately answers the question "within the last `20% * ` bytes worth of accessed blocks, what blocks were accessed enough to where loading them would make sense"?~
~For L5, we could use the same ghost cache, or we could use just the asynchronous loading mechanism to avoid latency spikes in individual reads.~
See #2709 and #5136 for more data and thinking on this.
Jira issue: PEBBLE-1301
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.