cockroachdb / cockroachdb/pebble

db: holistically consider space amplification compaction heuristics

Open
#5,005 2 comments 0 reactions 0 assignees View on GitHub
A-storage T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

We have a number of ways in which we may accumulate space amplification.

- Uncompacted tombstones and the KV pairs they delete may coexist within the LSM.
- Virtualized sstables pin backing sstables, with unreferenced sstable data being wastage.
- Blob files may contain unreferenced value data that is wastage.

Depending on the source of the space amplification, we have a few ways of reducing it:

- Delete-only compactions can reclaim disk space quickly and at the highest compaction priority level.
- Regular compaction picking inflates file sizes by both tombstone deleted bytes estimates and a linearly scaled [inflation for virtual sstables](https://github.com/cockroachdb/pebble/blob/3b5564003c0127984029ebc26cb9a1b6ed015938/compaction_picker.go#L1234-L1264).
- Blob file rewrite compactions can elide unreferenced values in a blob file by rewriting the blob file.

But there are some deficiencies,

- Blob file rewrites cannot elide data that is referenced by a backing sstable but not actually reachable through any of the extant virtual sstables. (#4915)
- Compactions could be better about deciding when to naturally rewrite blob files if doing so would reclaim disk space. (#4763)
- With the exception of delete-only compactions, reclaiming disk space is performed at the lowest compaction priority to avoid risking LSM inversion. We've observed situations in which extremely large volumes of space amplification are possible as these low-priority compactions are never run. (cockroachlabs/support#3264)

Space amplification is a barrier to increasing node density. We should consider space amplification holistically and adjust our compaction picking and mechanics to ensure we reclaim disk space promptly and without impacting user workloads.

A motivating slack conversation: https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1750862146318639

Jira issue: PEBBLE-526

Epic PEBBLE-1211

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with compaction_picker.go, especially the linked inflation logic, and read issues #4915 and #4763 for the known blob-file limitations. Review the listed compaction mechanisms and supporting discussion before defining a coherent scope; done would mean space amplification is reclaimed promptly without impacting user workloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.