apache / apache/datafusion

Proposal: Change `Accumulator` trait to accept `RecordBatch` / `num_rows` to allow faster `Count`

Open
#8,067 4 comments 2 reactions 0 assignees View on GitHub
api change enhancement performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently the `CountAccumulator` implementation requires `values: &[ArrayRef]` to be passed.

In order to eliminate scanning a (first) column, we need to be able to accept a `RecordBatch` or `num_rows` instead of `values: &[ArrayRef]`.

### Describe the solution you'd like

Rather than changing every method to accept a `RecordBatch` (and needing to update the code), I propose adding two new methods:

`update_record_batch(&mut self, recordbatch: &RecordBatch)`
`retract_record_batch(&mut self, recordbatch: &RecordBatch)`

The default implementation of the methods can use `update_batch` and `update_record_batch` (i.e. assume having at least one column).

In the aggregation code, we call `update_record_batch`/`retract_record_batch` instead.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the Accumulator trait, CountAccumulator, and the aggregation code that calls update_batch and retract_batch. Trace how RecordBatch and num_rows are available at those call sites; the work is done when record-batch updates are supported and Count can avoid scanning a first column without changing existing accumulator behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.