apache / apache/arrow

pyarrow.parquet.read_table : filtering a date column with a string version of a timestamp works with lists ('in' and 'not in' operators) but not with single values (=, >, <)

Open
#45,598 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hello,
with the following example :
```
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
import pandas as pd

df = pd.DataFrame({'date' : pd.date_range(start = pd.to_datetime('2025-02-01'),end = pd.to_datetime('2025-02-20'))})
df.to_parquet('dates.parquet')
timestamp_filter = pd.Timestamp('2025-02-18')
string_filter = str(timestamp_filter)
```
**'=' works with the Timestamp but not the string :**
```
In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','=',timestamp_filter)]
)
print(filtered_df)
Out:
pyarrow.Table
date: timestamp[ns]
----
date: [[2025-02-18 00:00:00.000000000]]

In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','=',string_filter)]
)
Out:
ArrowNotImplementedError: Function 'equal' has no kernel matching input types (timestamp[ns], string)

```
**however 'in' works with both the Timestamp and the string :**
```
In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','in',[timestamp_filter])]
)
print(filtered_df)
Out:
pyarrow.Table
date: timestamp[ns]
----
date: [[2025-02-18 00:00:00.000000000]]

In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','in',[string_filter])]
)
print(filtered_df)
Out:
pyarrow.Table
date: timestamp[ns]
----
date: [[2025-02-18 00:00:00.000000000]]
```

```
print(pd.__version__)
print(pa.__version__)
2.2.3
17.0.0
```
My problem is that my filter is stored in JSON format (in a dash.store component https://dash.plotly.com/dash-core-components/store); which stringifies pd.Timestamp values.
It would make it easier to manage if the single value operators '=' , '>', '<' could behave like the list like operators 'in' and 'not in'.

Many thanks,
Rgds

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.