apache / apache/paimon

[Feature] 1-1 mapping between paimon buckets and kafka partitions

Open
#3,249 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

When creating an Append for Message Queue table, as depicted in the screenshot here:
![Screenshot 2024-04-23 at 8 33 05 AM](https://github.com/apache/paimon/assets/23555517/d23bf690-7c06-4f90-91da-a4009cbbbb2b)

we can notice the following:
1. 5 buckets are specified, but unless data comes in the bucket is not created
2. If you check the Kafka partitions; partition 3 has keys 2, 3 and 4
3. These keys though end up in different buckets
4. Paimon does a shuffle, even though the parallelism is the same because it doesn't do 1-1 mapping

Because it is a Kafka-like message queue functionality, some users are confused, as they expect the same partitioning to happen and overall have a 1-1 mapping, between a Kafka partition and a paimon bucket.

At the same time, I believe this is a really good enhancement and should also allow to remove the shuffle between the operators, thus improving performance.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how Append message-queue tables assign Kafka partitions to Paimon buckets, including the shuffle between operators. Compare the current partition and bucket assignment behavior with the requested 1-1 mapping, then define completion as consistent partition-to-bucket routing without the unnecessary shuffle and with coverage for the reported five-bucket case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.