Proposal: Change `Accumulator` trait to accept `RecordBatch` / `num_rows` to allow faster `Count`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Currently the `CountAccumulator` implementation requires `values: &[ArrayRef]` to be passed.
In order to eliminate scanning a (first) column, we need to be able to accept a `RecordBatch` or `num_rows` instead of `values: &[ArrayRef]`.
### Describe the solution you'd like
Rather than changing every method to accept a `RecordBatch` (and needing to update the code), I propose adding two new methods:
`update_record_batch(&mut self, recordbatch: &RecordBatch)`
`retract_record_batch(&mut self, recordbatch: &RecordBatch)`
The default implementation of the methods can use `update_batch` and `update_record_batch` (i.e. assume having at least one column).
In the aggregation code, we call `update_record_batch`/`retract_record_batch` instead.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by locating the Accumulator trait, CountAccumulator, and the aggregation code that calls update_batch and retract_batch. Trace how RecordBatch and num_rows are available at those call sites; the work is done when record-batch updates are supported and Count can avoid scanning a first column without changing existing accumulator behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100