[Feature] Improve Universal Compaction
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Now, for universal compaction, when the condition for size amplification are met, [we select all the input sorted run to compaction](https://github.com/apache/incubator-paimon/blob/edaac231dbccce9cddd48937e37c74e84eb638f0/paimon-core/src/main/java/org/apache/paimon/mergetree/compact/UniversalCompaction.java#L104). , which is high cost, we can reduce the cost of the compression by selecting partial sorted runs.
Dividing one compaction process into multiple sub compaction will make the entire compaction process smoother
A simple optimization would be to compact so that just enough files are merged into the bottommost level (Lmax) to satisfy condition. It would work if we only need to pick some files from Lmax-1 or other level.
refer to. https://rocksdb.org/blog/2021/04/12/universal-improvements.html
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with paimon-core/src/main/java/org/apache/paimon/mergetree/compact/UniversalCompaction.java, especially the selection logic around line 104, and read the linked RocksDB universal compaction improvements. Define how partial sorted-run selection and multiple sub-compactions should satisfy the size-amplification condition while reducing compaction cost; the issue names no tests, so identify the relevant compaction coverage before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100