apache / apache/paimon

[Bug] spark fixed bucket write causes task data process unevenly for pk table

Open
#3,651 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8

### Compute Engine

spark

### Minimal reproduce step

When we use fixed bucket to write data for pk table, we set 1000 buckets, and want to process data with 1000 partitions evenly. But we found some tasks process no data and some tasks process many data. This will cause the overall running time of the task to be longer.

We found the root cause is `repartitionByExpression` will perform hash calculation again based on the bucket column. That will result in uneven distribution of partitions.

Some our configs:
```sql
primary-key = uuid
bucket = 1000
spark.sql.shuffle.partitions=1000
```
![image](https://github.com/apache/paimon/assets/135721692/8e35fa86-ade6-4817-ad0b-d5699d5b2055)

### What doesn't meet your expectations?

partitions data distribution.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reproducing the fixed-bucket primary-key table case with bucket=1000 and spark.sql.shuffle.partitions=1000, then inspect the repartitionByExpression path identified in the report. Done means the 1000 processing partitions receive data evenly without extending the overall task runtime.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.