[Improvement] Support removing user-specified repartition before writing when using zorder
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
In our company, users will use the repartition hint to merge small files, and the number of partitions in the repetition may not be reasonable. This will result in the zorder compression rate not being high enough, or because the number of repartition partitions is too small, resulting in a long sort time overhead. So we hope to have a configuration to remove the user-specified repartition.
### How should we improve?
Add a resolve rule to remove user-specified repartition before writing when using zorder
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
The issue names no files or tests; start by locating the resolve-rule area and the write path used for zorder. Determine how user-specified repartition is represented, then add the requested configuration behavior and verify that zorder writes remove it without changing other writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100