IcebergMetadataTableProvider ignores `projection` in `scan()`: projected/aggregate queries on populated metadata tables fail
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
main (`d1b8eac`)
### Describe the bug
The datafusion integration's metadata table provider ignores the `projection` argument in `TableProvider::scan()` (`crates/integrations/datafusion/src/table/metadata_table.rs`): it always returns an `IcebergMetadataScan` that reports the full metadata-table schema and emits unprojected batches. (`limit` is also unused, but that's only a missed optimization; filters never reach `scan()` since `supports_filters_pushdown` isn't overridden — projection is the correctness issue.)
This goes unnoticed because existing tests only query *empty* metadata tables with `SELECT *` (`test_metadata_table` in `integration_datafusion_test.rs`; the `$snapshots` checks in `insert_into.slt` run before any snapshot exists). Same class as #828, which fixed this for `IcebergTableScan`.
### To Reproduce
Any populated metadata table — e.g. against the seeded table in the `crates/sqllogictest` harness:
```sql
INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b');
SELECT count(*) FROM default.default.test_partitioned_table$snapshots;
-- DataFusion error: Internal error: Physical input schema should be the same as the
-- one converted from logical input schema. Differences:
-- - Different number of fields: (physical) 6 vs (logical) 0.
SELECT operation FROM default.default.test_partitioned_table$snapshots;
-- returns all 6 snapshot columns instead of just `operation`
```
### Expected behavior
`scan()` honors `projection`: make `IcebergMetadataScan` projection-aware (report the projected schema in its plan properties and project the emitted batches), mirroring how `IcebergTableScan` handles projection.
### Willingness to contribute
Yes — happy to submit the fix with sqllogictest regression coverage on populated metadata tables.
Contributor guide
Research direction
Start in crates/integrations/datafusion/src/table/metadata_table.rs and compare IcebergMetadataScan with IcebergTableScan's projection handling. Then inspect test_metadata_table in integration_datafusion_test.rs and the metadata-table cases in insert_into.slt, adding coverage for a populated table and projected or aggregate queries. Done means SELECT count(*) succeeds and SELECT operation returns only the requested column.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100