Extend partition statistics support to unpartitioned tables
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Proposed Change
Currently, the partition statistics file is only supported for partitioned tables.
https://github.com/apache/iceberg/blob/b987e60bbd581d6e9e583107d5a85022261ff0d8/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java#L207
While this design makes sense since it’s conceptually a “partition” statistics file, the information it contains is also valuable for unpartitioned tables.
For example, the Trino Iceberg connector currently needs to read manifest files ([TableStatisticsReader](https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TableStatisticsReader.java)) to build internal table statistics. If Iceberg provided a statistics file for unpartitioned tables as well, Trino (and other query engines) could leverage it directly, improving planning performance by avoiding expensive manifest reads.
“Extending partition statistics support” is just one possible approach. We are open to other options as well.
### Proposal document
_No response_
### Specifications
- [x] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
Contributor guide
Research direction
Start with core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java, especially the partitioned-table handling near the referenced line, and review how Trino's TableStatisticsReader currently reads manifests. Determine an approach for making equivalent statistics available to unpartitioned tables; done means query engines can use those statistics without expensive manifest reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100