apache / apache/arrow

[Python] read parquet in pyarrow is not idempotent for time period types

Open
#28,474 10 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

When reading a parquet file (attached) with a period type column via the "read_table" method, it returns "int64" on the first read. After applying "to_pandas" to the pyarrow table, subsequent "read_table" calls of the same parquet file in the same **Python session** return "ArrowPeriodType"
```java

import pyarrow
import pyarrow.parquet

pq_table = pyarrow.parquet.read_table("s3://my-bucket/my-prefix/period.parquet")
print(pq_table.schema.types)
# Out[1]: [DataType(int64)]

print(pq_table.to_pandas())
# Out[2]:
# col
# 0 2010-01

pq_table = pyarrow.parquet.read_table("s3://my-bucket/my-prefix/period.parquet")
print(pq_table.schema.types)
# Out[3]: [ArrowPeriodType(DataType(int64))]

pq_table = pyarrow.parquet.read_table("s3://my-bucket/my-prefix/period.parquet")
print(pq_table.schema.types)
# Out[4]: [ArrowPeriodType(DataType(int64))]
```
 

**Reporter**: [Abderrahmane Jaidi](https://issues.apache.org/jira/browse/ARROW-12732)
#### Original Issue Attachments:
- [period.parquet](https://issues.apache.org/jira/secure/attachment/13025261/period.parquet)

**Note**: *This issue was originally created as [ARROW-12732](https://issues.apache.org/jira/browse/ARROW-12732). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the behavior with the attached period.parquet file by calling parquet.read_table, to_pandas, and read_table again in one Python session. Trace the Python parquet-reading and pandas-conversion entry points to identify why the schema changes after conversion; done means repeated reads retain the same period-column type.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.