apache / apache/iceberg

Extend partition statistics support to unpartitioned tables

Open
#14,375 5 comments 3 reactions 0 assignees View on GitHub
proposal
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Proposed Change

Currently, the partition statistics file is only supported for partitioned tables.

https://github.com/apache/iceberg/blob/b987e60bbd581d6e9e583107d5a85022261ff0d8/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java#L207

While this design makes sense since it’s conceptually a “partition” statistics file, the information it contains is also valuable for unpartitioned tables.

For example, the Trino Iceberg connector currently needs to read manifest files ([TableStatisticsReader](https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TableStatisticsReader.java)) to build internal table statistics. If Iceberg provided a statistics file for unpartitioned tables as well, Trino (and other query engines) could leverage it directly, improving planning performance by avoiding expensive manifest reads.

“Extending partition statistics support” is just one possible approach. We are open to other options as well.

### Proposal document

_No response_

### Specifications

- [x] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start with core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java, especially the partitioned-table handling near the referenced line, and review how Trino's TableStatisticsReader currently reads manifests. Determine an approach for making equivalent statistics available to unpartitioned tables; done means query engines can use those statistics without expensive manifest reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.