apache / apache/auron

[TPCH Q4] Native Shuffle Size is bigger than vallina Spark's Shuffle size (155G VS 94G)

Open
#1,782 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

The query is TPCH Q4. The following is Vallina Spark:

Image

The following is Auron:

Image

Two issues here:

* Native shuffle size is bigger than vallina shuffle size: 155G VS 94G
* For the Native Plan, the shuffle write time from native side and java side differs a lot: 1.37h VS 16.7m

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start by reproducing TPCH Q4 with the native and vanilla Spark plans, then trace the native and Java shuffle metrics to identify why shuffle size and write time diverge. Done means explaining the discrepancy and adding a regression test or documented evidence for the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.