Timezone aware parquet read with schema and filters
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 94
Description
The parquet.read_table() method in pyarrow 8.0.0 added `schema` parameter which is great for handling timestamps, i.e., they are correctly converted from UTC to the timezone specified in the schema.
However, when `schema` is used together with `filters`, timezone conversion fails with "Cannot compare timestamp with timezone to timestamp without timezone" error. This was tested on 2 files created with different versions of spark. The test code, files and the output are attached.
**Reporter**: [Blaž Zupančič](https://issues.apache.org/jira/browse/ARROW-17058)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-17058) / @rok
#### Original Issue Attachments:
- [output.txt](https://issues.apache.org/jira/secure/attachment/13046683/output.txt)
- [pyarrow_bug.py](https://issues.apache.org/jira/secure/attachment/13046684/pyarrow_bug.py)
- [spark_parquet.py](https://issues.apache.org/jira/secure/attachment/13046685/spark_parquet.py)
- [spark-3.1.parquet](https://issues.apache.org/jira/secure/attachment/13046681/spark-3.1.parquet)
- [spark-3.2.parquet](https://issues.apache.org/jira/secure/attachment/13046682/spark-3.2.parquet)
**Note**: *This issue was originally created as [ARROW-17058](https://issues.apache.org/jira/browse/ARROW-17058). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the attached pyarrow_bug.py reproducer against the two attached Spark Parquet files and inspect its output. Trace the parquet.read_table() path when schema and filters are both supplied; done means timezone-aware schema conversion works without the timestamp comparison error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100