[CH] AQE seems not working on skewed partitions
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
SQL: d_1075_0.sql
The data itself is skewed on column `type` about 65% values on one type. Vanila works fine, 45s max processing time. But gluten with CH backend seems not working about 1.76 hour the biggest task.
vanila:
gluten:
Contributor guide
Research direction
Start with the d_1075_0.sql workload and compare the vanilla and Gluten executions, focusing on AQE handling of the skewed type partition with the ClickHouse backend. Reproduce the task-time difference and determine why the largest task reaches about 1.76 hours instead of the reported 45 seconds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala, spark
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100