Bug(arrow-row): calling `convert_raw` function cause "offset overflow" panic
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Describe the bug**
- Datafusion Table Info:
Having a `http_url` column, which DataType is `Utf8`, and it has a lot of distinct values.
- Datafusion SQL
```sql
select http_url from tab group http_url
```
- Panic Info
- Reason
`GroupValuesRows` in Datafusion stores rows using `Rows`, and this bug may be triggered when calling `emit` function.
https://github.com/apache/arrow-rs/blob/49e714de6e951169d0d5e73381af247ad0230fcf/arrow-row/src/variable.rs#L217-L226
In the extreme case, if I `append` two `Utf8` values which size are large(len1 + len2 > i32::MAX) into Rows twice, and then call `convert_rows`, which should also trigger the bug. 🤔
**To Reproduce**
**Expected behavior**
**Additional context**
Contributor guide
Research direction
Start in arrow-row/src/variable.rs around the referenced lines and trace convert_raw, convert_rows, emit, and Rows. Compare the described large Utf8 append scenario with DataFusion's GroupValuesRows path. Done means the offset overflow panic is prevented for the described case and the behavior is verified with a reproducible regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100