apache / apache/arrow

Using pandas Period in filters for parquet fails

Open
#48,023 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

```python
import pandas as pd
# Ensure `ArrowPeriodType` is registered
import pandas.core.arrays.arrow.extension_types

df = pd.DataFrame(
{
"month": pd.period_range("2024-01", "2024-12", freq="M"),
"data": range(12),
}
)
df.to_parquet("test.parquet", engine="pyarrow")

pd.read_parquet(
"test.parquet",
engine="pyarrow",
filters=[("month", ">=", pd.Period("2024-07", freq="M"))]
)
# Could not convert Period('2024-07', 'M') with type Period: did not
# recognize Python value type when inferring an Arrow data type
```

A similar example using `pd.Timestamp` succeeds. It's not clear to me if something on the pandas side needs to be done here.

Original pandas report: https://github.com/pandas-dev/pandas/issues/62769

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.