apache / apache/datafusion-ballista
Add PartialReduce operator / optimization
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Partial-reduce applies a second agreggation to already aggregated results, often significantly reducing the shuffle size for aggregations - significantly reducing shuffle write / read costs.
**Describe the solution you'd like**
Add partial reduce to the stages that further reduces aggregation.
This can be implemented by applying a repartioning + partial reduce in the same stage.
A threshold probably has to be added to avoid slowing down almost-unique inputs.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by tracing the aggregation stages and the repartitioning and shuffle path described in the issue. Determine where a second aggregation can be applied and how a threshold should avoid slowing down nearly unique inputs; done means aggregation stages reduce shuffle data without regressing those inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100