[Python] read parquet in pyarrow is not idempotent for time period types
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
When reading a parquet file (attached) with a period type column via the "read_table" method, it returns "int64" on the first read. After applying "to_pandas" to the pyarrow table, subsequent "read_table" calls of the same parquet file in the same **Python session** return "ArrowPeriodType"
```java
import pyarrow
import pyarrow.parquet
pq_table = pyarrow.parquet.read_table("s3://my-bucket/my-prefix/period.parquet")
print(pq_table.schema.types)
# Out[1]: [DataType(int64)]
print(pq_table.to_pandas())
# Out[2]:
# col
# 0 2010-01
pq_table = pyarrow.parquet.read_table("s3://my-bucket/my-prefix/period.parquet")
print(pq_table.schema.types)
# Out[3]: [ArrowPeriodType(DataType(int64))]
pq_table = pyarrow.parquet.read_table("s3://my-bucket/my-prefix/period.parquet")
print(pq_table.schema.types)
# Out[4]: [ArrowPeriodType(DataType(int64))]
```
**Reporter**: [Abderrahmane Jaidi](https://issues.apache.org/jira/browse/ARROW-12732)
#### Original Issue Attachments:
- [period.parquet](https://issues.apache.org/jira/secure/attachment/13025261/period.parquet)
**Note**: *This issue was originally created as [ARROW-12732](https://issues.apache.org/jira/browse/ARROW-12732). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the behavior with the attached period.parquet file by calling parquet.read_table, to_pandas, and read_table again in one Python session. Trace the Python parquet-reading and pandas-conversion entry points to identify why the schema changes after conversion; done means repeated reads retain the same period-column type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100