apache / apache/paimon

[Feature] Append table support distribution mode before write to reduce small files

Open
#3,844 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Currently Paimon's append table is written directly by Flink writer operator subtask which will cause too many small files. We can set some shuffle conditions before writing to reduce the generation of small files, especially stream processing.Such as key by partition.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue concerns Paimon's append-table writes from a Flink writer operator and proposes adding distribution before writing, but it names no files, tests, or concrete implementation. First map the append-table write path and existing shuffle or partitioning behavior; done should include an agreed distribution design that reduces small files without breaking stream-processing writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.