apache / apache/arrow

Inconsistency between written Parquet and Arrow schemas when `coerce_timestamps` is set

Open
#47,336 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Arrow's parquet writer does not update the `unit` of timestamp columns in the written base64 encoded `ARROW::schema` that have been casted using the `coerce_timestamps` argument. This leads to an inconsistent behavior/confusion when reading such a file as the two schemas (Parquet and ARROW) contain different timestamp units.

Arrow's parquet reader uses the timestamp units from the Parquet schema and only extracts timezone info from the ARROW::schema if available whereas the parquet reader in Polars directly uses the ARROW::schema when available ignoring the Parquet schema

### Expected Behavior
The parquet writer may also update the timestamp unit for the casted columns in the ARROW::schema for consistency and/or the parquet reader should override the unit from ARROW::schema over Parquet schema (faithful reconstruction of units + timezone?).

### More context

https://github.com/pola-rs/polars/issues/21392#issuecomment-3186570001

## Related Issues

https://github.com/pola-rs/polars/issues/21392 and likely https://github.com/pola-rs/polars/issues/23949

Thank you

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.