cockroachdb / cockroachdb/pebble
disagg: make cache persistent
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
As a follow-up to #2541, make the cache persistent, using the following ideas:
- we persist the mapping of which logical blocks (in objects) map to which physical blocks (in the cache); we don't persist any other cache eviction metadata
- use a "working set" of blocks (say 100 of them). Whenever a block gets evicted, we don't immediately overwrite/reuse the block but use one of the working set blocks instead. This allows us to delay flushing of the metadata, which is still valid as long as the original block isn't overwritten. After we flush the metadata, the blocks that were evicted become part of the new working set.
- metadata changes can be logged efficiently using the following scheme:
- consider the metadata to be a table where the `i`-th entry describes the logical contents of the `i`-th cache block; the table is fixed in size (say N blocks).
- we maintain a global "pointer" `P` to an entry in this table
- whenever we make a change to the table, we append an entry with the change, and an entry that logs the current value of entry `P`; `P` is advanced circularly (i.e. `P = (P+1)%N`)
- note that the entire contents of the table are always available from the last `2N` entries; we can have a circular log of fixed size `2N` with no need for background log rotations / compactions
Jira issue: PEBBLE-191
Epic CRDB-40358
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.