[TPCH Q4] Native Shuffle Size is bigger than vallina Spark's Shuffle size (155G VS 94G)
Open
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 22
Description
The query is TPCH Q4. The following is Vallina Spark:
The following is Auron:
Two issues here:
* Native shuffle size is bigger than vallina shuffle size: 155G VS 94G
* For the Native Plan, the shuffle write time from native side and java side differs a lot: 1.37h VS 16.7m
Contributor guide
Research direction
No source file, test, or entry point is named. Start by reproducing TPCH Q4 with the native and vanilla Spark plans, then trace the native and Java shuffle metrics to identify why shuffle size and write time diverge. Done means explaining the discrepancy and adding a regression test or documented evidence for the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100