Add option to prune column stats when rewriting manifests
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Feature Request / Improvement
By default Iceberg infers metrics for up to 100 columns (`write.metadata.metrics.max-inferred-column-defaults`), each with `truncate(16)` bounds. On wide tables this generates a large amount of per-data-file column stats, bloating manifests and slowing query planning.
Users can tighten this by lowering the default or setting `write.metadata.metrics.column.*` to `none`/`counts` — but changing the config does **not** shrink existing manifests. `RewriteManifests` currently copies each data file's stats verbatim, so the now-excess stats persist until every affected data file is itself rewritten (expensive, and often unnecessary just to reclaim metadata space).
Proposal: add a `pruneColumnStats()` option to the `RewriteManifests` API that drops column stats exceeding the table's current metrics config as manifests are rewritten:
- columns configured `none` → all stats dropped
- columns configured `counts` → bounds dropped
- columns configured `truncate(n)` / `full` → unchanged (existing bounds are never re-truncated)
Pruning applies only to files in manifests that are actually rewritten. When no clustering function is set via `clusterBy`, enabling it triggers a full rewrite so all data files are regrouped.
This lets users reclaim manifest space and speed up planning after reducing metrics collection, without rewriting the data files themselves.
### Query engine
None
Contributor guide
Research direction
Start at the RewriteManifests API and its manifest-rewrite implementation, then trace how the table’s current metrics configuration and clusterBy behavior are handled. Locate the existing RewriteManifests tests and verify pruning for none, counts, truncate/full, rewritten manifests only, and the full-rewrite case when no clustering function is set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100