apache / apache/arrow

[Python] Error of wrong number of rows read from Parquet file

Open
#33,265 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

on version greater than 6.0.1 fail to read tables saying expected length n, got n=1 rows

 

Tables can be read column by column fine, or with a fixed number of rows matching the meta data fine.      Reads correctly in version 6.0.1

**Environment**: Python Info
3.10.7 (tags/v3.10.7:6cc6b13, Sep 5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)]
Pyarrow Info
6.0.1
Platform Info
Windows-10-10.0.19042-SP0
Windows
10
10.0.19042
19042
AMD64
**Reporter**: [Blake erickson](https://issues.apache.org/jira/browse/ARROW-18064)
#### Original Issue Attachments:
- [badplug.parquet](https://issues.apache.org/jira/secure/attachment/13050964/badplug.parquet)
- [readBadParquet.py](https://issues.apache.org/jira/secure/attachment/13050963/readBadParquet.py)
- [screenshot-1.png](https://issues.apache.org/jira/secure/attachment/13051179/screenshot-1.png)

**Note**: *This issue was originally created as [ARROW-18064](https://issues.apache.org/jira/browse/ARROW-18064). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the attached readBadParquet.py against badplug.parquet and compare the reported row count across the stated PyArrow versions. Inspect the attached reproduction and screenshot to establish the failing read path; done means reproducing the mismatch, identifying its cause, and adding a regression test for the affected Parquet read behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.