apache / apache/paimon

[Feature] Optimize splits generator for unaware bucket mode

Open
#3,235 0 comments 0 reactions 1 assignee Claimed by @Aitozi View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

In stream mode, the unaware bucket mode will use the `FIFOSplitAssigner`, So we could avoid Bin-Packing serveral files into one split. IMO, it has two benefit:

- In the `FIFOSplitAssigner`, we have a work-stealing mechanism that allows for higher total throughput as the split size decreases.
- We can avoid the issue of skewed files by comparing two files with similar row counts but very different file sizes.

### Solution

One file one split for unaware bucket mode

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.