Improved performance for streaming grouping with single string columns
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Follow on to https://github.com/apache/arrow-datafusion/issues/7064
The `GroupsValues` for aggregates need to handle "emitTo" for streaming groups so that the can flush groups that have already been built but will never be seen again.
The initial implementation of the specialized accumulator for Uft8/LargeUtf8 in #8827 is inefficient in that it copies / rehashes any strings remaining in the set after emission
This is likely not a large performance overhead in practice as most groups should be emitted so only a few groups will need to be rehashed. However, if it turns out it is a problem, we can make something more optimized
### Describe the solution you'd like
Optimize emitTo for binary groups
https://github.com/apache/arrow-datafusion/pull/9188
### Describe alternatives you've considered
I have one proposal in https://github.com/apache/arrow-datafusion/pull/9188 (look at ArrowStringSet::emit_first_n) -- it works and passes tests but I think is very complicated and hard to convince onesself that the `unsafe` usage is sound
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the GroupsValues streaming-group handling and the ArrowStringSet::emit_first_n proposal in PR 9188, along with the follow-on context from issue 7064. Compare the proposed approach with the existing Utf8/LargeUtf8 accumulator and its tests. Done means optimizing emitTo for binary groups while preserving the passing test behavior and avoiding unsound unsafe usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100