Inconsistency between written Parquet and Arrow schemas when `coerce_timestamps` is set
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Arrow's parquet writer does not update the `unit` of timestamp columns in the written base64 encoded `ARROW::schema` that have been casted using the `coerce_timestamps` argument. This leads to an inconsistent behavior/confusion when reading such a file as the two schemas (Parquet and ARROW) contain different timestamp units.
Arrow's parquet reader uses the timestamp units from the Parquet schema and only extracts timezone info from the ARROW::schema if available whereas the parquet reader in Polars directly uses the ARROW::schema when available ignoring the Parquet schema
### Expected Behavior
The parquet writer may also update the timestamp unit for the casted columns in the ARROW::schema for consistency and/or the parquet reader should override the unit from ARROW::schema over Parquet schema (faithful reconstruction of units + timezone?).
### More context
https://github.com/pola-rs/polars/issues/21392#issuecomment-3186570001
## Related Issues
https://github.com/pola-rs/polars/issues/21392 and likely https://github.com/pola-rs/polars/issues/23949
Thank you
### Component(s)
C++, Parquet
Contributor guide
Assessment
This issue has not been assessed yet.