apache / apache/iceberg-rust

IcebergMetadataTableProvider ignores `projection` in `scan()`: projected/aggregate queries on populated metadata tables fail

Open
#2,819 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

main (`d1b8eac`)

### Describe the bug

The datafusion integration's metadata table provider ignores the `projection` argument in `TableProvider::scan()` (`crates/integrations/datafusion/src/table/metadata_table.rs`): it always returns an `IcebergMetadataScan` that reports the full metadata-table schema and emits unprojected batches. (`limit` is also unused, but that's only a missed optimization; filters never reach `scan()` since `supports_filters_pushdown` isn't overridden — projection is the correctness issue.)

This goes unnoticed because existing tests only query *empty* metadata tables with `SELECT *` (`test_metadata_table` in `integration_datafusion_test.rs`; the `$snapshots` checks in `insert_into.slt` run before any snapshot exists). Same class as #828, which fixed this for `IcebergTableScan`.

### To Reproduce

Any populated metadata table — e.g. against the seeded table in the `crates/sqllogictest` harness:

```sql
INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b');

SELECT count(*) FROM default.default.test_partitioned_table$snapshots;
-- DataFusion error: Internal error: Physical input schema should be the same as the
-- one converted from logical input schema. Differences:
-- - Different number of fields: (physical) 6 vs (logical) 0.

SELECT operation FROM default.default.test_partitioned_table$snapshots;
-- returns all 6 snapshot columns instead of just `operation`
```

### Expected behavior

`scan()` honors `projection`: make `IcebergMetadataScan` projection-aware (report the projected schema in its plan properties and project the emitted batches), mirroring how `IcebergTableScan` handles projection.

### Willingness to contribute

Yes — happy to submit the fix with sqllogictest regression coverage on populated metadata tables.

Contributor guide

Open the contributing guide

Research direction

Start in crates/integrations/datafusion/src/table/metadata_table.rs and compare IcebergMetadataScan with IcebergTableScan's projection handling. Then inspect test_metadata_table in integration_datafusion_test.rs and the metadata-table cases in insert_into.slt, adding coverage for a populated table and projected or aggregate queries. Done means SELECT count(*) succeeds and SELECT operation returns only the requested column.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.