apache / apache/paimon

[Feature] Support to include partition in the outputPartition

Open
#5,293 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

In spark SPJ, it supports to report the partition key as the output partition. We could leverage this to increase the parallelism when working with cross partition join.

For example:

t1:

```
CREATE TABLE t1 (id INT, dt STRING, app STRING) PARTITIONED BY (dt, app) TBLPROPERTIES ('bucket'='10', 'bucket-key' = 'id')
```

t2:

```
CREATE TABLE t2 (id INT, dt STRING, app STRING) PARTITIONED BY (dt, app) TBLPROPERTIES ('bucket'='10', 'bucket-key' = 'id')
```

```
SELECT * FROM t1 JOIN t2 on t1.id = t2.id and t1.app = t2.app where dt = '20250316'
```

If we only take the bucket into account, it only could run with the 10 parallelism limited by the bucket. We could let the `app` also make up the output partition, which will increase the parallelism and bring better performance

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by tracing the Spark SPJ handling of outputPartition and compare it with the partitioned-table join example; done means partition keys such as app can contribute to the output partition and increase parallelism for the described cross-partition join.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.