[Feature] Append table support distribution mode before write to reduce small files
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Currently Paimon's append table is written directly by Flink writer operator subtask which will cause too many small files. We can set some shuffle conditions before writing to reduce the generation of small files, especially stream processing.Such as key by partition.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue concerns Paimon's append-table writes from a Flink writer operator and proposes adding distribution before writing, but it names no files, tests, or concrete implementation. First map the append-table write path and existing shuffle or partitioning behavior; done should include an agreed distribution design that reduces small files without breaking stream-processing writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100