apache / apache/iceberg-rust

expose statistics in iceberg-datafusion for fast COUNT()

Open
#1,428 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Running `SELECT COUNT(1)` when using iceberg-datafusion results in a table scan. This can be avoided by implementing [ExecutionPlan::statistics](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.statistics). Datafusion does this for its built-in parquet scanner by fetching the statistics from parquet metadata when [constructing the ExecutionPlan](https://github.com/apache/datafusion/blob/367331e4a83412f450fa4b7b41aad6dc3624d04b/datafusion/core/src/datasource/listing/table.rs#L914). I was looking to implement this in a similar way (at least for tables without deletes) by iterating over the `ManifestEntry`s and summing the `record_count`s. I have a draft PR but wanted to confirm this approach is acceptable before putting in the work to clean it up.

### Describe the solution you'd like

`count(*)` in datafusion does not perform a table scan

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Research direction

Start with DataFusion's ExecutionPlan::statistics entry point and the linked listing/table.rs construction example. Then inspect iceberg-datafusion's manifest handling and ManifestEntry record_count usage, especially the stated no-deletes case. Done means COUNT(*) avoids a table scan while preserving correct statistics behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.