apache / apache/iceberg

Column Stats Improvements

Open
#13,153 1 comment 3 reactions 0 assignees View on GitHub
Iceberg V4 not-stale proposal
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

## Motivation
Column statistics are currently stored as a mapping of field id to values across multiple columns (lower/upper bounds, value/nan/null counts, sizes). This storage model has critical limitations as the number of columns increases and as new types are being added to Iceberg:
Inefficient Storage due to map-based structure:
* Large memory overhead during planning/processing
* Inability to project specific stats (e.g., only null_value_counts for column X)
* Type Erasure: Original logical/physical types are lost when stored as binary blobs, causing:
* Lossy type inference during reads
* Schema evolution challenges (e.g., widening types)
* Rigid Schema: Stats are tied to the data_file entry record, limiting extensibility for new stats.

## Goals
Improve the column stats representation to allow for the following:
* Projectability: Enable independent access to specific stats (e.g., lower_bounds without loading upper_bounds).
* Type Preservation: Store original data types to support accurate reads and schema evolution.
* Flexible/Extensible Representation: Allow per-field stats structures (e.g., complex types like Geo/Variant).

## Non-Goals
The following issues are out-of-scope or impractical to address
* Supporting unlimited stats for tables with extreme column counts
* Addressing Parquet column amplification in manifest files

### Proposal document

https://s.apache.org/iceberg-column-stats

### Specifications

- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start with the linked column-stats proposal and the checked Table and REST specifications; the issue names no source files, tests, or implementation entry points. Done requires an agreed representation that provides projectable, type-preserving, extensible column statistics for the in-scope specifications.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.