apache / apache/datafusion

Operators should respect batch_size instead of Emit::All

Open
#19,481 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

In many operators, when the input is done, the output is emitted all at once, this leads to huge batches, which can be inefficient in some cases, and OOM in other cases.

For example:
Using a fair pool implementation where each operator gets 1GB of memory, and an aggregate's total size is 0.99GB.
The aggregate will Emit::All, and output the 0.99GB batch, and a following sort will immediately try and allocate memory for the sorting(let's say 2x the batch size).
The memory pool has no idea that the agg is done at this point, as the reservation still exists, so it will try and maintain 1GB per operator.

The sort now attempts to get about 1.98GB, which is almost twice what it is allocated.

### Describe the solution you'd like

If the aggregate(and any other operator, for that matter) respected batch_size, it would output smaller batches, which are easily handled by ExternalSorter, which will spill when it needs to, and at some point when poll returns Read(None) we can drop the input stream, and the reservation will drop and the sort will get even more memory.
That will make every application much more resilient under every memory constraints.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing operator output handling around batch_size and Emit::All, especially aggregate output, ExternalSorter, and poll returning Read(None). Identify the affected operators and verify that output is split into manageable batches while preserving the intended memory-release behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.