[Bug] spark fixed bucket write causes task data process unevenly for pk table
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8
### Compute Engine
spark
### Minimal reproduce step
When we use fixed bucket to write data for pk table, we set 1000 buckets, and want to process data with 1000 partitions evenly. But we found some tasks process no data and some tasks process many data. This will cause the overall running time of the task to be longer.
We found the root cause is `repartitionByExpression` will perform hash calculation again based on the bucket column. That will result in uneven distribution of partitions.
Some our configs:
```sql
primary-key = uuid
bucket = 1000
spark.sql.shuffle.partitions=1000
```

### What doesn't meet your expectations?
partitions data distribution.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by reproducing the fixed-bucket primary-key table case with bucket=1000 and spark.sql.shuffle.partitions=1000, then inspect the repartitionByExpression path identified in the report. Done means the 1000 processing partitions receive data evenly without extending the overall task runtime.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100