apache / apache/arrow-rs

Bug(arrow-row): calling `convert_raw` function cause "offset overflow" panic

Open
#6,112 6 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**

- Datafusion Table Info:
Having a `http_url` column, which DataType is `Utf8`, and it has a lot of distinct values.

- Datafusion SQL

```sql
select http_url from tab group http_url
```

- Panic Info
image

- Reason
`GroupValuesRows` in Datafusion stores rows using `Rows`, and this bug may be triggered when calling `emit` function.
https://github.com/apache/arrow-rs/blob/49e714de6e951169d0d5e73381af247ad0230fcf/arrow-row/src/variable.rs#L217-L226

In the extreme case, if I `append` two `Utf8` values which size are large(len1 + len2 > i32::MAX) into Rows twice, and then call `convert_rows`, which should also trigger the bug. 🤔

**To Reproduce**

**Expected behavior**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in arrow-row/src/variable.rs around the referenced lines and trace convert_raw, convert_rows, emit, and Rows. Compare the described large Utf8 append scenario with DataFusion's GroupValuesRows path. Done means the offset overflow panic is prevented for the described case and the behavior is verified with a reproducible regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.