apache / apache/iceberg

Implement metrics evaluators that work directly with ContentStats

Open
#16,218 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

### Problem

Followup from PR [feedback](https://github.com/apache/iceberg/pull/16100#discussion_r3185164312).

The content file adapters convert `ContentStats` to v3 style bounds. Each call allocates a new map and, for bounds serializes every field value into a fresh `ByteBuffer`.

This is acceptable when a single evaluator calls each method once per file, but if multiple evaluators are used (e.g. `InclusiveMetricsEvaluator` + `StrictMetricsEvaluator`), the maps are rebuilt and discarded repeatedly.

### Fix

Add metrics evaluators that work directly with `ContentStats` / `FieldStats` rather than going through the `ContentFile` bounds/counts maps. This avoids the intermediate map allocation and per-field serialization.

Once ContentStats-aware evaluators are available, ensure they are used in all v4 code paths (scan planning, manifest filtering, etc.) instead of falling back to the adapter's map-based methods.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing metrics evaluators, ContentStats/FieldStats adapters, and the v4 scan-planning and manifest-filtering paths. Done means evaluators read ContentStats directly and all v4 paths use them instead of the adapter's map-based bounds and counts methods.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.