apache / apache/arrow-rs

Parquet Field Display Drops Timestamp Precision

Open
#6,949 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

`convert_timestamp_millis_to_string` and `convert_timestamp_micros_to_string` round the timestamp down to seconds, thus losing any fractional seconds when converting the timestamp to JSON:

https://github.com/apache/arrow-rs/blob/4f1f6e57c568fae8233ab9da7d7c7acdaea4112a/parquet/src/record/api.rs#L926

Wouldn't it be better to print the fractional seconds in the RFC3339 format using `%.3f` for millis and `%.6f` for micros, like so:

```
format!("{}", dt.format("%Y-%m-%d %H:%M:%S%.3f %:z"))
```

This is a simple fix, and I am happy to create a PR, but was wondering if there's any reason for the current approach, which throws away fractional seconds.

Contributor guide

Open the contributing guide

Research direction

Start in parquet/src/record/api.rs around line 926 and inspect convert_timestamp_millis_to_string and convert_timestamp_micros_to_string. Confirm that JSON output preserves fractional seconds with the appropriate millisecond or microsecond precision, then verify the affected behavior with the relevant project tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.