apache / apache/datafusion

Track long-term replacement for destructive EmitTo::First in aggregation

Open
#23,251 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Problem

#23178 and #23249 both found regressions caused by using `EmitTo::First(batch_size)` during hash aggregate terminal output.

The short-term fixes materialize aggregate output once with `EmitTo::All` and then slice the materialized `RecordBatch`. This avoids repeated `EmitTo::First` work for final and partial terminal output paths.

However, this does not solve the broader API/design problem:

- `EmitTo::First(n)` sounds like cheap prefix emission, but it also mutates internal state.
- It removes emitted groups and renumbers remaining group indexes.
- It can require expensive copying, shifting, allocation, and lookup-state maintenance.
- It is easy to misuse in terminal output paths where no more groups will be interned.
- It also makes `GroupValues` and `GroupsAccumulator` implementations harder to maintain.

### Related discussion

In #23249, @Rachelint suggested it may be better to eliminate `EmitTo::First` entirely, including sorted cases, because it is expensive, hard to optimize, and difficult to support.

@2010YOUY01 suggested that after blocked state management, the ideal shape may be:

- no `EmitTo::First`
- O(1) `EmitTo::FirstBlock`

### Goal

Track the long-term design for replacing or narrowing `EmitTo::First` usage in aggregation.

### Non-goals

This issue is not intended to block the short-term regression fixes in #23178 or #23249.

Those fixes can continue using materialize-once-and-slice. This issue tracks the broader follow-up design and incremental migration path.

Contributor guide

Open the contributing guide

Research direction

Start by reading the aggregation code around EmitTo::First, GroupValues, and GroupsAccumulator, then review #23178 and #23249 for the current regression fixes and design discussion. Define a replacement or narrower migration path, while keeping the materialize-once-and-slice fixes independent; done means the long-term direction and incremental steps are documented and agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.