cockroachdb / cockroachdb/pebble

implement a bloom filter cache for L6

Open
#5,655 1 comment 0 reactions 1 assignee Claimed by @RaduBerinde View on GitHub
A-storage C-performance P-1 T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

We want to make use of L6 bloom filters, but these blocks can be big (much bigger than data blocks) and for large LSMs are unlikely to fit in the block cache. Using them without care could cause significant degradation of random workloads.

We want to find blocks that are actually accessed frequently enough to make up for the overhead of loading them. To achieve this we can implement a special cache that tracks filter block cache misses for L6 (or L5 and L6).

~The specific proposal is to use a byte-budgeted FIFO queue (for each block cache shard). The queue contains just filter block references, and is sized so that the total block sizes in the queue is below e.g. 20% of the block cache size. Each entry has a counter that is incremented every time we tried to access that block. Once the count reaches a threshold (e.g. ` / `), we asynchronously load that block into the block cache. The ghost cache approximately answers the question "within the last `20% * ` bytes worth of accessed blocks, what blocks were accessed enough to where loading them would make sense"?~

~For L5, we could use the same ghost cache, or we could use just the asynchronous loading mechanism to avoid latency spikes in individual reads.~

See #2709 and #5136 for more data and thinking on this.

Jira issue: PEBBLE-1301

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.