apache / apache/iceberg

Add option to prune column stats when rewriting manifests

Open
#17,257 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

By default Iceberg infers metrics for up to 100 columns (`write.metadata.metrics.max-inferred-column-defaults`), each with `truncate(16)` bounds. On wide tables this generates a large amount of per-data-file column stats, bloating manifests and slowing query planning.

Users can tighten this by lowering the default or setting `write.metadata.metrics.column.*` to `none`/`counts` — but changing the config does **not** shrink existing manifests. `RewriteManifests` currently copies each data file's stats verbatim, so the now-excess stats persist until every affected data file is itself rewritten (expensive, and often unnecessary just to reclaim metadata space).

Proposal: add a `pruneColumnStats()` option to the `RewriteManifests` API that drops column stats exceeding the table's current metrics config as manifests are rewritten:
- columns configured `none` → all stats dropped
- columns configured `counts` → bounds dropped
- columns configured `truncate(n)` / `full` → unchanged (existing bounds are never re-truncated)

Pruning applies only to files in manifests that are actually rewritten. When no clustering function is set via `clusterBy`, enabling it triggers a full rewrite so all data files are regrouped.

This lets users reclaim manifest space and speed up planning after reducing metrics collection, without rewriting the data files themselves.

### Query engine

None

Contributor guide

Open the contributing guide

Research direction

Start at the RewriteManifests API and its manifest-rewrite implementation, then trace how the table’s current metrics configuration and clusterBy behavior are handled. Locate the existing RewriteManifests tests and verify pruning for none, counts, truncate/full, rewritten manifests only, and the full-rewrite case when no clustering function is set.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.