expose statistics in iceberg-datafusion for fast COUNT()
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
Running `SELECT COUNT(1)` when using iceberg-datafusion results in a table scan. This can be avoided by implementing [ExecutionPlan::statistics](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.statistics). Datafusion does this for its built-in parquet scanner by fetching the statistics from parquet metadata when [constructing the ExecutionPlan](https://github.com/apache/datafusion/blob/367331e4a83412f450fa4b7b41aad6dc3624d04b/datafusion/core/src/datasource/listing/table.rs#L914). I was looking to implement this in a similar way (at least for tables without deletes) by iterating over the `ManifestEntry`s and summing the `record_count`s. I have a draft PR but wanted to confirm this approach is acceptable before putting in the work to clean it up.
### Describe the solution you'd like
`count(*)` in datafusion does not perform a table scan
### Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Contributor guide
Research direction
Start with DataFusion's ExecutionPlan::statistics entry point and the linked listing/table.rs construction example. Then inspect iceberg-datafusion's manifest handling and ManifestEntry record_count usage, especially the stated no-deletes case. Done means COUNT(*) avoids a table scan while preserving correct statistics behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100