facebook / facebook/rocksdb

Request for a stricter limit on worst-case space-amplification

Open
#9,561 1 comment 0 reactions 0 assignees View on GitHub
enhancement feature-request up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

This is a feature request.

### Expected behavior

With respect to worst-case space-amplification:
- Leveled compaction in theory has an implied limit of 1 + 1/plf where plf is the per-level fanout. But see #9423 to understand why in practice the limit is larger and the results below show that.
- Universal doesn't have a limit. There is [max_size_amplification_percent](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/universal_compaction.h#L48) but that isn't a limit, it just determines when major compaction is triggered. Universal can also benefit from [level0_file_num_compaction_trigger](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/options.h#L237) but that is also not a strict limit in practice. ScyllaDB has an optional space-amp goal ([SAG](https://www.scylladb.com/product/scylla-enterprise/2021-x/)) for size-tiered.
- Integrated BlobDB has [blob_garbage_collection_force_threshold](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/advanced_options.h#L905) and [blob_garbage_collection_age_cutoff](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/advanced_options.h#L892). While these have a strong impact on worst-case space-amp it isn't clear yet that they provide a strict limit.

### Actual behavior

Full details were provided to the team (sorry, bad OSS behavior) and I used 2 workloads with db_bench

- in-memory - use fillseq to load 40M KV pairs, run readwhilewriting for 30 minutes, then run overwrite with no writer rate limit for 1 hour
- IO-bound - same as in-memory, except fillseq loads 800M KV pairs

Then I compared the size at the end of readrandom vs the max size of database directory during overwrite and I will list the ratio of max / initial size here for the in-memory workload.

- 2.0 for leveled
- 5.6 for universal
- 3.5 for blob with a naive config
- 23.2 for blob with a better config. This is too large and I need to revisit the problem to try and explain it.

The max/initial size ratio for the IO-bound workload using buffered IO and then O_DIRECT

- 1.2 and 1.4 for leveled
- 3.8 and failed for universal - by failed I mean the overwrite test failed when the 3T database directory became full. At that point the space-amp was ~6X
- 1.4 and 1.3 for blob with the naive config
- 1.7 and 1.7 for blob with the better config

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.