apache / apache/iceberg-rust

StaticTableProvider can't be reconstructed in another process

Open
#3,017 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

`IcebergStaticTableProvider` holds a `Table`, which owns a live `FileIO` and is
bound to the catalog it was loaded from. Neither survives serialization.

That blocks distributed execution: engines like DataFusion Ballista plan on one
node and execute on others, so `IcebergTableScan` has to be shipped to workers
that then rebuild storage before they can read data files. The provider knows
which catalog it loaded from, but drops that information, leaving consumers no
way to identify the table well enough to rebuild it elsewhere.

The repo already has the shape needed: `CatalogBuilder::load` takes `name` and
`props`, and `iceberg-catalog-loader` selects a builder by `type`. That triple
just isn't recorded on the read path.

### Describe the solution you'd like

Add `IcebergCatalogConfig { type, name, props }` to `iceberg-datafusion`: plain
data, no live connections, mirroring the loader's inputs. Let
`IcebergStaticTableProvider` record it and expose it alongside `table_ident()`
and `snapshot_id()`, and let `IcebergTableScan` carry it into the physical plan.

Together those are enough to rebuild the catalog, load the `Table`, and pin the
same snapshot. Nothing in the crate would connect using them; the config is an
`Option` defaulting to `None`, so existing behavior is unchanged.

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start in the iceberg-datafusion crate by tracing IcebergStaticTableProvider, IcebergTableScan, and the physical-plan path, then compare the catalog inputs used by CatalogBuilder::load and iceberg-catalog-loader. The work is complete when the plain catalog configuration is exposed, carried through planning, and existing behavior remains unchanged when it is absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.