apache / apache/arrow

[Parquet][C++][Python] "List index overflow" when read parquet file

Open
#33,188 14 comments 4 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

From issue https://github.com/apache/arrow/issues/14229.

The bug looks like this:
- create a pandas dataframe with **one column** and `n` rows, `n < max(int32)`
- each elemenet is a list with `m` integers, `m * n > max(int32)`
- save to a parquet file
- reading from the parquet file fails with "OSError: List index overflow"

See comment below on details to reproudce this bug:
https://github.com/apache/arrow/issues/14229#issuecomment-1272223773

Tested with a small dataset, the error might come from below code.
https://github.com/apache/arrow/blob/master/cpp/src/parquet/level_conversion.cc#L63-L64
`OffsetType` is `int32`, but the loop is executed (and `*offset` is incremented) `m * n` times which is beyond `max(int32)`.

**Reporter**: [Yibo Cai](https://issues.apache.org/jira/browse/ARROW-17983) / @cyb70289

**Note**: *This issue was originally created as [ARROW-17983](https://issues.apache.org/jira/browse/ARROW-17983). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with cpp/src/parquet/level_conversion.cc at lines 63-64 and review the reproduction details in the linked comment on issue 14229. Reproduce the failure with a pandas dataframe whose nested-list element count exceeds the int32 limit, then verify that reading the resulting Parquet file no longer reports "List index overflow".

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.