cockroachdb / cockroachdb/pebble
*: defense in depth against large keys
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
Large keys that end up in the manifest as boundaries (start, end keys) can cause the manifest to balloon. There exists a threshold ([`MaxManifestFileSize`](https://github.com/cockroachdb/pebble/blob/1fdc5c0536fd810a356d153c127359c3a3698fb9/options.go#L598-L601)) after which any compaction, flush, ingestion that causes a version edit will result in a rotation and rewrite of the entire manifest file. This can put manifest writes into an `O(n)` regime (rewriting the entire manifest each time), rather than the expected ~O(1) regime (i.e. incremental updates).
There are likely some improvements we can make in this area that would bolster our defense in depth.
Jira issue: PEBBLE-99
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with MaxManifestFileSize in options.go and trace how compaction, flush, and ingestion create version edits and trigger manifest rotation. Inspect the manifest-writing paths and existing coverage for large boundary keys; done means oversized keys cannot cause repeated full rewrites and incremental manifest updates remain effective.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100