apache / apache/arrow-rs

Better support for reading and writing UTC adjusted time types in parquet

Open
#8,027 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

In parquet, the [TIME](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time) logical type has a boolean parameter for whether the values are adjusted to UTC. Arrow doesn't have any notion of time zones for time types. #6278 added a metadata key whose presence indicates that the parquet logical TIME type should be adjusted to UTC. This solution is limited in that it is not documented and the metadata key is not added on the read side unless the file was written by arrow-rs and the reader doesn't skip the arrow metadata.

**Describe the solution you'd like**

I would like this feature to be documented and for the key to be added automatically whenever the parquet TIME type is adjusted to UTC.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing the metadata-key implementation from #6278 through the parquet TIME read and write paths. Check how UTC adjustment is represented when Arrow metadata is skipped, then document the convention and add the key automatically for adjusted TIME values. Done means both reading and writing consistently preserve this information and coverage verifies the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.