[Feature] Support to include partition in the outputPartition
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
In spark SPJ, it supports to report the partition key as the output partition. We could leverage this to increase the parallelism when working with cross partition join.
For example:
t1:
```
CREATE TABLE t1 (id INT, dt STRING, app STRING) PARTITIONED BY (dt, app) TBLPROPERTIES ('bucket'='10', 'bucket-key' = 'id')
```
t2:
```
CREATE TABLE t2 (id INT, dt STRING, app STRING) PARTITIONED BY (dt, app) TBLPROPERTIES ('bucket'='10', 'bucket-key' = 'id')
```
```
SELECT * FROM t1 JOIN t2 on t1.id = t2.id and t1.app = t2.app where dt = '20250316'
```
If we only take the bucket into account, it only could run with the 10 parallelism limited by the bucket. We could let the `app` also make up the output partition, which will increase the parallelism and bring better performance
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Start by tracing the Spark SPJ handling of outputPartition and compare it with the partitioned-table join example; done means partition keys such as app can contribute to the output partition and increase parallelism for the described cross-partition join.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100