apache / apache/parquet-java

Emit and Read min/max statistics for int96 timestamp columns

Open
#3,242 3 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

### Describe the enhancement requested

Parquet-java does not emit or read stats for int96 timestamp columns. Since int96 is used as the default timestamp in spark, this limits a lot of optimization opportunities. Engines like [Photon](https://www.databricks.com/product/photon) populate the statistics for the int96 timestamps correctly. So parquet-java coud also emit the statistics, and also allow reading these statistics from known good writers.

### Component(s)

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing parquet-java's existing statistics handling for timestamp columns and compare the expected int96 min/max representation with the behavior of known-good writers such as Photon. The work is done when parquet-java emits and reads correct min/max statistics for int96 timestamp columns, with coverage for both paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.