Refresh schema information in iceberg datafusion catalog provider
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
Currently our implementation of [IcebergCatalogProvider](https://github.com/apache/iceberg-rust/blob/5692c0d051c9faa273f4cca80e9b49943ada863b/crates/integrations/datafusion/src/catalog.rs#L34) captures a snapshot of schema information at create time. This is incorrect since catalog will change. We should have a procedure to refresh schema information at background.
Another option is to use [`AsyncCatalogListProvider`](https://docs.rs/datafusion/latest/datafusion/catalog/trait.AsyncCatalogProviderList.html), but it requires using low level api, see https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/remote_catalog.rs
Contributor guide
Research direction
Start in crates/integrations/datafusion/src/catalog.rs at IcebergCatalogProvider and inspect how schema information is captured during creation. Compare the background-refresh approach with DataFusion's AsyncCatalogListProvider and the remote_catalog example. Done means catalog schema changes are refreshed rather than remaining a creation-time snapshot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100