apache / apache/datafusion

Improved performance for streaming grouping with single string columns

Open
#9,195 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Follow on to https://github.com/apache/arrow-datafusion/issues/7064

The `GroupsValues` for aggregates need to handle "emitTo" for streaming groups so that the can flush groups that have already been built but will never be seen again.

The initial implementation of the specialized accumulator for Uft8/LargeUtf8 in #8827 is inefficient in that it copies / rehashes any strings remaining in the set after emission

This is likely not a large performance overhead in practice as most groups should be emitted so only a few groups will need to be rehashed. However, if it turns out it is a problem, we can make something more optimized

### Describe the solution you'd like

Optimize emitTo for binary groups

https://github.com/apache/arrow-datafusion/pull/9188

### Describe alternatives you've considered

I have one proposal in https://github.com/apache/arrow-datafusion/pull/9188 (look at ArrowStringSet::emit_first_n) -- it works and passes tests but I think is very complicated and hard to convince onesself that the `unsafe` usage is sound

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the GroupsValues streaming-group handling and the ArrowStringSet::emit_first_n proposal in PR 9188, along with the follow-on context from issue 7064. Compare the proposed approach with the existing Utf8/LargeUtf8 accumulator and its tests. Done means optimizing emitTo for binary groups while preserving the passing test behavior and avoiding unsound unsafe usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.