StaticTableProvider can't be reconstructed in another process
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
`IcebergStaticTableProvider` holds a `Table`, which owns a live `FileIO` and is
bound to the catalog it was loaded from. Neither survives serialization.
That blocks distributed execution: engines like DataFusion Ballista plan on one
node and execute on others, so `IcebergTableScan` has to be shipped to workers
that then rebuild storage before they can read data files. The provider knows
which catalog it loaded from, but drops that information, leaving consumers no
way to identify the table well enough to rebuild it elsewhere.
The repo already has the shape needed: `CatalogBuilder::load` takes `name` and
`props`, and `iceberg-catalog-loader` selects a builder by `type`. That triple
just isn't recorded on the read path.
### Describe the solution you'd like
Add `IcebergCatalogConfig { type, name, props }` to `iceberg-datafusion`: plain
data, no live connections, mirroring the loader's inputs. Let
`IcebergStaticTableProvider` record it and expose it alongside `table_ident()`
and `snapshot_id()`, and let `IcebergTableScan` carry it into the physical plan.
Together those are enough to rebuild the catalog, load the `Table`, and pin the
same snapshot. Nothing in the crate would connect using them; the config is an
`Option` defaulting to `None`, so existing behavior is unchanged.
### Willingness to contribute
I can contribute to this feature independently
Contributor guide
Research direction
Start in the iceberg-datafusion crate by tracing IcebergStaticTableProvider, IcebergTableScan, and the physical-plan path, then compare the catalog inputs used by CatalogBuilder::load and iceberg-catalog-loader. The work is complete when the plain catalog configuration is exposed, carried through planning, and existing behavior remains unchanged when it is absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100