apache / apache/iceberg

Handle struct-level metrics from non-Parquet writers

Open
#17,463 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

https://github.com/apache/iceberg/pull/17412 removed struct-level metrics from `TestInclusiveMetricsEvaluator` to align with the current behavior of Parquet. However, the spec doesn't technically prevent a writer from emitting struct-level metrics for v1-v3. Also the v4 content stats don't keep track of struct-level metrics.
The goal here is to investigate how we want to deal with struct-level metrics in general and especially if non-Parquet writers emit them

### Query engine

None

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by reading TestInclusiveMetricsEvaluator and the changes in pull request 17412, then compare the specification's v1-v3 rules with v4 content stats. Investigate how struct-level metrics from non-Parquet writers should be handled; done means the project has an agreed behavior for these metrics, backed by appropriate evaluator coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.