apache / apache/datafusion-ballista
Optimize shuffle before coalesce
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
This looks inefficient. We are writing lots of shuffle files, reading them, and coalescing them into a single partition. Can we do the coalesce step before the shuffle write in this case?

**Describe the solution you'd like**
Optimize
**Describe alternatives you've considered**
None
**Additional context**
None
Contributor guide
Research direction
Start by tracing the query path that writes shuffle files and then coalesces them into a single partition, using the issue description and attached opt-coalesce diagram as context. Done means establishing whether coalescing before the shuffle write is correct and measurably avoids the unnecessary shuffle files and reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100