[Parquet][C++][Python] "List index overflow" when read parquet file
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
From issue https://github.com/apache/arrow/issues/14229.
The bug looks like this:
- create a pandas dataframe with **one column** and `n` rows, `n < max(int32)`
- each elemenet is a list with `m` integers, `m * n > max(int32)`
- save to a parquet file
- reading from the parquet file fails with "OSError: List index overflow"
See comment below on details to reproudce this bug:
https://github.com/apache/arrow/issues/14229#issuecomment-1272223773
Tested with a small dataset, the error might come from below code.
https://github.com/apache/arrow/blob/master/cpp/src/parquet/level_conversion.cc#L63-L64
`OffsetType` is `int32`, but the loop is executed (and `*offset` is incremented) `m * n` times which is beyond `max(int32)`.
**Reporter**: [Yibo Cai](https://issues.apache.org/jira/browse/ARROW-17983) / @cyb70289
**Note**: *This issue was originally created as [ARROW-17983](https://issues.apache.org/jira/browse/ARROW-17983). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with cpp/src/parquet/level_conversion.cc at lines 63-64 and review the reproduction details in the linked comment on issue 14229. Reproduce the failure with a pandas dataframe whose nested-list element count exceeds the int32 limit, then verify that reading the resulting Parquet file no longer reports "List index overflow".
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100