[EPIC] (Even More) Grouping / Group By / Aggregation Performance
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Aggregation is a key operation of Analytic engines. DataFusion has made great progress recently (e.g. https://github.com/apache/arrow-datafusion/issues/4973 and https://github.com/apache/arrow-datafusion/issues/6889)
This Epic gathers other potential ways we can improve the performance of aggregation
Core Hash Grouping Algorithm:
- [x] https://github.com/apache/arrow-datafusion/issues/6969
- [x] https://github.com/apache/arrow-datafusion/issues/7064
- [x] https://github.com/apache/arrow-datafusion/issues/9403
- [x] https://github.com/apache/arrow-datafusion/issues/7023
- [x] https://github.com/apache/arrow-datafusion/issues/7095
Specialized Aggregators:
- [x] https://github.com/apache/arrow-datafusion/issues/6906
- [x] https://github.com/apache/arrow-datafusion/issues/5547
- [x] https://github.com/apache/arrow-datafusion/issues/5472
- [ ] https://github.com/apache/arrow-datafusion/issues/7065
- [ ] https://github.com/apache/arrow-datafusion/issues/7066
New features:
- [x] https://github.com/apache/arrow-datafusion/issues/6937
- [ ] https://github.com/apache/arrow-datafusion/issues/9562
- [ ] https://github.com/apache/arrow-datafusion/issues/8699
- [x] https://github.com/apache/arrow-datafusion/issues/8934
Improved partitioning:
- [ ] https://github.com/apache/arrow-datafusion/issues/6928
- [ ] https://github.com/apache/arrow-datafusion/issues/7001
- [ ] https://github.com/apache/arrow-datafusion/issues/6822
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
The payload names no files, tests, or entry points. Start by opening the unchecked linked issues 7065, 7066, 9562, 8699, 6928, 7001, and 6822; completion must follow the selected issue's scope and acceptance criteria rather than this umbrella epic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100