[Improvement] Support local sort in the Spark's insertRepartitionBeforeWrite optimization rule
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Currently, the Spark's `insertRepartitionBeforeWrite` optimization rule will be skipped when logical plans are `Sort` regardless of whether they are local or not: https://github.com/apache/kyuubi/blob/fa9e6be/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/RepartitionBeforeWritingBase.scala#L133.
It makes sense for global sort, since inserting repartition after the sort changes the semantics of the original plans and doing before that only introduces an additional shuffle.
However, inserting repartition before local sort will help to sort rebalanced partitions even if locally, and it aligns with the behavior of queries that explicitly use both `REPARTITION|REBALANCE` and `SORT BY`.
### How should we improve?
This issue proposes to support local sort in the Spark's `insertRepartitionBeforeWrite` optimization rule by inserting repartition before the sort.
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start at extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/RepartitionBeforeWritingBase.scala around line 133 and inspect how the rule handles local and global Sort plans. Done means local sorts support repartition insertion before the sort while global-sort semantics remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100