cockroachdb / cockroachdb/pebble

disagg: make cache persistent

Open
#2,542 5 comments 0 reactions 0 assignees View on GitHub
A-storage T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

As a follow-up to #2541, make the cache persistent, using the following ideas:
- we persist the mapping of which logical blocks (in objects) map to which physical blocks (in the cache); we don't persist any other cache eviction metadata
- use a "working set" of blocks (say 100 of them). Whenever a block gets evicted, we don't immediately overwrite/reuse the block but use one of the working set blocks instead. This allows us to delay flushing of the metadata, which is still valid as long as the original block isn't overwritten. After we flush the metadata, the blocks that were evicted become part of the new working set.
- metadata changes can be logged efficiently using the following scheme:
- consider the metadata to be a table where the `i`-th entry describes the logical contents of the `i`-th cache block; the table is fixed in size (say N blocks).
- we maintain a global "pointer" `P` to an entry in this table
- whenever we make a change to the table, we append an entry with the change, and an entry that logs the current value of entry `P`; `P` is advanced circularly (i.e. `P = (P+1)%N`)
- note that the entire contents of the table are always available from the last `2N` entries; we can have a circular log of fixed size `2N` with no need for background log rotations / compactions

Jira issue: PEBBLE-191

Epic CRDB-40358

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.