[Feature] 1-1 mapping between paimon buckets and kafka partitions
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
When creating an Append for Message Queue table, as depicted in the screenshot here:

we can notice the following:
1. 5 buckets are specified, but unless data comes in the bucket is not created
2. If you check the Kafka partitions; partition 3 has keys 2, 3 and 4
3. These keys though end up in different buckets
4. Paimon does a shuffle, even though the parallelism is the same because it doesn't do 1-1 mapping
Because it is a Kafka-like message queue functionality, some users are confused, as they expect the same partitioning to happen and overall have a 1-1 mapping, between a Kafka partition and a paimon bucket.
At the same time, I believe this is a really good enhancement and should also allow to remove the shuffle between the operators, thus improving performance.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how Append message-queue tables assign Kafka partitions to Paimon buckets, including the shuffle between operators. Compare the current partition and bucket assignment behavior with the requested 1-1 mapping, then define completion as consistent partition-to-bucket routing without the unnecessary shuffle and with coverage for the reported five-bucket case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100