Column Stats Improvements
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
## Motivation
Column statistics are currently stored as a mapping of field id to values across multiple columns (lower/upper bounds, value/nan/null counts, sizes). This storage model has critical limitations as the number of columns increases and as new types are being added to Iceberg:
Inefficient Storage due to map-based structure:
* Large memory overhead during planning/processing
* Inability to project specific stats (e.g., only null_value_counts for column X)
* Type Erasure: Original logical/physical types are lost when stored as binary blobs, causing:
* Lossy type inference during reads
* Schema evolution challenges (e.g., widening types)
* Rigid Schema: Stats are tied to the data_file entry record, limiting extensibility for new stats.
## Goals
Improve the column stats representation to allow for the following:
* Projectability: Enable independent access to specific stats (e.g., lower_bounds without loading upper_bounds).
* Type Preservation: Store original data types to support accurate reads and schema evolution.
* Flexible/Extensible Representation: Allow per-field stats structures (e.g., complex types like Geo/Variant).
## Non-Goals
The following issues are out-of-scope or impractical to address
* Supporting unlimited stats for tables with extreme column counts
* Addressing Parquet column amplification in manifest files
### Proposal document
https://s.apache.org/iceberg-column-stats
### Specifications
- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
Contributor guide
Research direction
Start with the linked column-stats proposal and the checked Table and REST specifications; the issue names no source files, tests, or implementation entry points. Done requires an agreed representation that provides projectable, type-preserving, extensible column statistics for the in-scope specifications.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100