apache / apache/parquet-format

Add new optional type parameters Offset to TIMESTAMP

Open
#458 2 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

### Describe the enhancement requested

Hi, I'm new around here, please let me know if this request is better elsewhere.

I'd like to propose an optional type parameter called `Offset` to TIMESTAMP logical types.

In my common use case of Parquet files, the data is a running log with many rows, such that any one row group is unlikely to have more than a few days at a time.

The idea of the `Offset` parameter would be to store for each row group (in Int64) an offset from Unix epoch, then the data would be stored relative to that offset.

This provides a couple of benefits:

1. row groups could be selectively downsized (when possible) to INT32 physical types. This could save significant amounts of file size if I understand correctly. At millisecond level accuracy, INT32 could support row groups up to ~48 days long.[^1]
2. [The docs](https://github.com/apache/parquet-format/blob/4f208158dba80ff4bff4afaa4441d7270103dff6/LogicalTypes.md?plain=1#L458) identify that all TIMESTAMPs, but particularly those with NANOs accuracy have range limitations due to the INT64 limitation. Adding an `Offset` would allow practically unlimited ranges for TIMESTAMPs.

[^1]: with an offset set in the middle of row group values, given the signed nature of INT32

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the TIMESTAMP logical type section in LogicalTypes.md, especially its current range limitations and physical-type representation. Evaluate how an optional Offset parameter would be represented per row group and how it would affect INT32 and INT64 storage. Done would require a decided format design and corresponding specification updates.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.