[Bug] KyuubiSparkSQLExtension will cause shuffle write data size expand too much
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
1、kyuubi1.7.1 、kyuubi-extension-spark-3-3_2.12-1.7.1.jar (download from maven)、spark 3.3.3
2、insert into ... select ... statment
3、if I user AQE and KyuubiSparkSQLExtension, the Exchange node is RoundRobinPartitioning, and shuffle write data size expand very large, I finally cancel the query
4、if I not use KyuubiSparkSQLExtension
5、I'm I get the wrong way to use KyuubiSparkSQLExtension?
### Affects Version(s)
1.7.1
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
```yaml
##AQE Adaptive Query Execution
spark.sql.adaptive.enabled true
spark.sql.adaptive.forceApply false
spark.sql.adaptive.logLevel info
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.adaptive.fetchShuffleBlocksInBatch true
spark.sql.adaptive.localShuffleReader.enabled true
spark.sql.adaptive.skewJoin.enabled true
spark.sql.adaptive.skewJoin.skewedPartitionFactor 5
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes 400MB
spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin 0.2
spark.sql.adaptive.autoBroadcastJoinThreshold 100MB
spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold 400MB
spark.sql.autoBroadcastJoinThreshold -1
#spark.sql.extensions org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.apache.kyuubi.sql.KyuubiSparkSQLExtension
spark.sql.extensions org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start by reproducing the reported INSERT INTO ... SELECT behavior on Kyuubi 1.7.1 with Spark 3.3.3, AQE enabled, and KyuubiSparkSQLExtension enabled and disabled. Compare the Exchange node and shuffle write size using the supplied engine configuration; done means identifying whether the extension causes the excessive shuffle and documenting or fixing the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark, sql
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100