Handle struct-level metrics from non-Parquet writers
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
https://github.com/apache/iceberg/pull/17412 removed struct-level metrics from `TestInclusiveMetricsEvaluator` to align with the current behavior of Parquet. However, the spec doesn't technically prevent a writer from emitting struct-level metrics for v1-v3. Also the v4 content stats don't keep track of struct-level metrics.
The goal here is to investigate how we want to deal with struct-level metrics in general and especially if non-Parquet writers emit them
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by reading TestInclusiveMetricsEvaluator and the changes in pull request 17412, then compare the specification's v1-v3 rules with v4 content stats. Investigate how struct-level metrics from non-Parquet writers should be handled; done means the project has an agreed behavior for these metrics, backed by appropriate evaluator coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100