apache / apache/paimon

[Feature] Improve Universal Compaction

Open
#1,769 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Now, for universal compaction, when the condition for size amplification are met, [we select all the input sorted run to compaction](https://github.com/apache/incubator-paimon/blob/edaac231dbccce9cddd48937e37c74e84eb638f0/paimon-core/src/main/java/org/apache/paimon/mergetree/compact/UniversalCompaction.java#L104). , which is high cost, we can reduce the cost of the compression by selecting partial sorted runs.

Dividing one compaction process into multiple sub compaction will make the entire compaction process smoother

A simple optimization would be to compact so that just enough files are merged into the bottommost level (Lmax) to satisfy condition. It would work if we only need to pick some files from Lmax-1 or other level.

refer to. https://rocksdb.org/blog/2021/04/12/universal-improvements.html

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with paimon-core/src/main/java/org/apache/paimon/mergetree/compact/UniversalCompaction.java, especially the selection logic around line 104, and read the linked RocksDB universal compaction improvements. Define how partial sorted-run selection and multiple sub-compactions should satisfy the size-amplification condition while reducing compaction cost; the issue names no tests, so identify the relevant compaction coverage before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.