Parquet Field Display Drops Timestamp Precision
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
`convert_timestamp_millis_to_string` and `convert_timestamp_micros_to_string` round the timestamp down to seconds, thus losing any fractional seconds when converting the timestamp to JSON:
https://github.com/apache/arrow-rs/blob/4f1f6e57c568fae8233ab9da7d7c7acdaea4112a/parquet/src/record/api.rs#L926
Wouldn't it be better to print the fractional seconds in the RFC3339 format using `%.3f` for millis and `%.6f` for micros, like so:
```
format!("{}", dt.format("%Y-%m-%d %H:%M:%S%.3f %:z"))
```
This is a simple fix, and I am happy to create a PR, but was wondering if there's any reason for the current approach, which throws away fractional seconds.
Contributor guide
Research direction
Start in parquet/src/record/api.rs around line 926 and inspect convert_timestamp_millis_to_string and convert_timestamp_micros_to_string. Confirm that JSON output preserves fractional seconds with the appropriate millisecond or microsecond precision, then verify the affected behavior with the relevant project tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100