Grouping operations on large datasets can overflow i32 offsets
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
The ByteGroupValueBuilder can raise an `offset overflow` error if the columns (string or binary) have a large number of values.
Also see: https://github.com/apache/datafusion-comet/issues/4718
### To Reproduce
https://github.com/apache/datafusion-comet/issues/4718 has a reproduction
### Expected behavior
Grouping operations ideally should not overflow and return an error, even on large input data
### Additional context
_No response_
Contributor guide
Research direction
Start with ByteGroupValueBuilder and use the reproduction in datafusion-comet issue 4718 to observe the offset overflow during grouping on large string or binary columns. Trace the grouping path involved and verify that large inputs no longer trigger the reported overflow, while preserving the expected error behavior described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100