cockroachdb / cockroachdb/pebble

*: defense in depth against large keys

Open
#1,741 15 comments 1 reaction 0 assignees View on GitHub
O-postmortem P-3 T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

Large keys that end up in the manifest as boundaries (start, end keys) can cause the manifest to balloon. There exists a threshold ([`MaxManifestFileSize`](https://github.com/cockroachdb/pebble/blob/1fdc5c0536fd810a356d153c127359c3a3698fb9/options.go#L598-L601)) after which any compaction, flush, ingestion that causes a version edit will result in a rotation and rewrite of the entire manifest file. This can put manifest writes into an `O(n)` regime (rewriting the entire manifest each time), rather than the expected ~O(1) regime (i.e. incremental updates).

There are likely some improvements we can make in this area that would bolster our defense in depth.

Jira issue: PEBBLE-99

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with MaxManifestFileSize in options.go and trace how compaction, flush, and ingestion create version edits and trigger manifest rotation. Inspect the manifest-writing paths and existing coverage for large boundary keys; done means oversized keys cannot cause repeated full rewrites and incremental manifest updates remain effective.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.