apache / apache/paimon

[Feature] Improve `compaction.max-sorted-run-num` implementation

Open
#988 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Currently the `compaction.max-sorted-run-num` parameter is basically useless, it only affects `UniversalCompaction` to pick the candidates for compaction, but `UniversalCompaction` must pick to level 1, so it doesn't play a limiting role.

We need to modify `CompactRewriter` so that it can limit its own maximum memory usage, for example by using iterative compaction.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how compaction.max-sorted-run-num flows from UniversalCompaction into CompactRewriter, focusing on why selection currently must reach level 1. Define and verify the intended maximum-memory behavior for CompactRewriter, including iterative compaction, and confirm that the parameter has a limiting effect rather than only changing candidate selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.