apache / apache/datafusion

Grouping operations on large datasets can overflow i32 offsets

Open
#23,694 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The ByteGroupValueBuilder can raise an `offset overflow` error if the columns (string or binary) have a large number of values.

Also see: https://github.com/apache/datafusion-comet/issues/4718

### To Reproduce

https://github.com/apache/datafusion-comet/issues/4718 has a reproduction

### Expected behavior

Grouping operations ideally should not overflow and return an error, even on large input data

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with ByteGroupValueBuilder and use the reproduction in datafusion-comet issue 4718 to observe the offset overflow during grouping on large string or binary columns. Trace the grouping path involved and verify that large inputs no longer trigger the reported overflow, while preserving the expected error behavior described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.