cockroachdb / cockroachdb/pebble

smart filter block loading

Open
#3,788 1 comment 0 reactions 1 assignee Claimed by @RaduBerinde View on GitHub
A-storage O-support P-3 T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

This is inspired from https://github.com/cockroachlabs/support/issues/2992 and https://github.com/cockroachdb/pebble/issues/3787

We can do a much better job at loading filter blocks. We currently either use them (blocking a Get) or don't use them at all. Ideally, we would load larger blocks that are valuable in the background.

Here's a proposal:
- blocks not in L6 and under a certain size (e.g. 128KB) are loaded inline with the query, like today
- larger blocks we do not load immediately but maintain a metadata-only cache for them, where we record how many times they would have been useful - i.e. how many `SeekPrefixGE` found no results with the prefix. Once we have enough "hits", we load the block in the background. The number of hits required should be proportional to the size (e.g. must have `size/128KB` recent "hits").
- ideally for the block cache, only the times where the block actually helped exclude something should be counted as a "hit"

https://github.com/cockroachdb/pebble/issues/3734 should also be kept in mind. We might eventually have multiple filter blocks of different sizes which we use in various combinations depending on level.

Jira issue: PEBBLE-226

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.