apache / apache/arrow

Timezone aware parquet read with schema and filters

Open
#32,366 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 20h
Merged PRs (30d)
94

Description

The parquet.read_table() method in pyarrow 8.0.0 added `schema` parameter which is great for handling timestamps, i.e., they are correctly converted from UTC to the timezone specified in the schema.

However, when `schema` is used together with `filters`, timezone conversion fails with "Cannot compare timestamp with timezone to timestamp without timezone" error. This was tested on 2 files created with different versions of spark. The test code, files and the output are attached.

**Reporter**: [Blaž Zupančič](https://issues.apache.org/jira/browse/ARROW-17058)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-17058) / @rok
#### Original Issue Attachments:
- [output.txt](https://issues.apache.org/jira/secure/attachment/13046683/output.txt)
- [pyarrow_bug.py](https://issues.apache.org/jira/secure/attachment/13046684/pyarrow_bug.py)
- [spark_parquet.py](https://issues.apache.org/jira/secure/attachment/13046685/spark_parquet.py)
- [spark-3.1.parquet](https://issues.apache.org/jira/secure/attachment/13046681/spark-3.1.parquet)
- [spark-3.2.parquet](https://issues.apache.org/jira/secure/attachment/13046682/spark-3.2.parquet)

**Note**: *This issue was originally created as [ARROW-17058](https://issues.apache.org/jira/browse/ARROW-17058). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the attached pyarrow_bug.py reproducer against the two attached Spark Parquet files and inspect its output. Trace the parquet.read_table() path when schema and filters are both supplied; done means timezone-aware schema conversion works without the timestamp comparison error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.