apache / apache/datafusion-comet
Improve performance for roundrobin shuffle writes for nested schemas
Open
area:shuffle
enhancement
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
Noticed dramatic slowdown for tasks with shuffle writes for RR repartition comparing to Spark, 2-3x to be more precise.
Specifically for the nested schemas, there is some inefficiency when native shuffle deals with wide/complex fields. its invisible for TPC-* benchmarks because of historical flat schema. But in real user world this is pretty much downside.
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.