pyarrow.parquet.read_table : filtering a date column with a string version of a timestamp works with lists ('in' and 'not in' operators) but not with single values (=, >, <)
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hello,
with the following example :
```
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
import pandas as pd
df = pd.DataFrame({'date' : pd.date_range(start = pd.to_datetime('2025-02-01'),end = pd.to_datetime('2025-02-20'))})
df.to_parquet('dates.parquet')
timestamp_filter = pd.Timestamp('2025-02-18')
string_filter = str(timestamp_filter)
```
**'=' works with the Timestamp but not the string :**
```
In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','=',timestamp_filter)]
)
print(filtered_df)
Out:
pyarrow.Table
date: timestamp[ns]
----
date: [[2025-02-18 00:00:00.000000000]]
In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','=',string_filter)]
)
Out:
ArrowNotImplementedError: Function 'equal' has no kernel matching input types (timestamp[ns], string)
```
**however 'in' works with both the Timestamp and the string :**
```
In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','in',[timestamp_filter])]
)
print(filtered_df)
Out:
pyarrow.Table
date: timestamp[ns]
----
date: [[2025-02-18 00:00:00.000000000]]
In:
filtered_df = pq.read_table(
source = 'dates.parquet',
filters = [('date','in',[string_filter])]
)
print(filtered_df)
Out:
pyarrow.Table
date: timestamp[ns]
----
date: [[2025-02-18 00:00:00.000000000]]
```
```
print(pd.__version__)
print(pa.__version__)
2.2.3
17.0.0
```
My problem is that my filter is stored in JSON format (in a dash.store component https://dash.plotly.com/dash-core-components/store); which stringifies pd.Timestamp values.
It would make it easier to manage if the single value operators '=' , '>', '<' could behave like the list like operators 'in' and 'not in'.
Many thanks,
Rgds
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.