apache / apache/arrow-rs

Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB)

Open
#7,973 20 comments 0 reactions 0 assignees View on GitHub
bug help wanted parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**
I can write a table with a column whose entries together are larger than 2**31-1.
Reading them fails with an overflow error.

**To Reproduce**
```
from pathlib import Path
import pyarrow
import tempfile
import arro3.io

schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
sizes = [5068563] * 500

new_table = pyarrow.Table.from_pylist([{"html": b"0" * size} for size in sizes], schema)

with tempfile.TemporaryDirectory() as tmp_dir:
tmp = Path(tmp_dir) / "evil.parquet"
arro3.io.write_parquet(new_table, tmp)
print(len(arro3.io.read_parquet(tmp).read_all()))
```
This throws the following error:
```
Traceback (most recent call last):
File "overflow.py", line 87, in
print(len(arro3.io.read_parquet(tmp).read_all()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: C Data interface error: Parquet argument error: Parquet error: index overflow decoding byte array
```

**Expected behavior**
Should print 500.

**Additional context**
arro3 is just a thin wrapper around arro-rs, which I'm using for convenience.
I'm using arro3-io version 0.5.1 and python 3.12.
arro3 uses arrow-rs 54.1.0 and our internal library uses arrow-rs 55.1.0. The problem is present with both.
pyarrow can read the file just fine, that's what makes me think that this might be an error in arrow-rs.

Contributor guide

Open the contributing guide

Research direction

Start by running the overflow.py reproduction through arro3.io.read_parquet and read_all, then trace the arrow-rs Parquet reader path that reports "index overflow decoding byte array." Compare the result with pyarrow on the same file and verify the fix by confirming that reading the 500-row file succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.