apache / apache/kyuubi

[Improvement] Support removing user-specified repartition before writing when using zorder

Open
#6,560 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

In our company, users will use the repartition hint to merge small files, and the number of partitions in the repetition may not be reasonable. This will result in the zorder compression rate not being high enough, or because the number of repartition partitions is too small, resulting in a long sort time overhead. So we hope to have a configuration to remove the user-specified repartition.

### How should we improve?

Add a resolve rule to remove user-specified repartition before writing when using zorder

### Are you willing to submit PR?

- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by locating the resolve-rule area and the write path used for zorder. Determine how user-specified repartition is represented, then add the requested configuration behavior and verify that zorder writes remove it without changing other writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.