apache / apache/iceberg-rust

Support `CREATE EXTERNAL TABLE` backed by a Catalog with DataFusion

Open
#2,021 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Background

DataFusion already supports CREATE EXTERNAL TABLE ... STORED AS ICEBERG. Today, iceberg-rust integrates via IcebergTableProviderFactory, but the factory primarily supports registering a static table (e.g., created from a metadata JSON path). That works for:
```sql
-- Static table (existing, backward compatible)
CREATE EXTERNAL TABLE my_table
STORED AS ICEBERG
LOCATION '/path/to/metadata.json';
```

However, we also want CREATE EXTERNAL TABLE to create a normal IcebergTableProvider backed by a Catalog, so users can define the catalog via SQL OPTIONS (and then resolve tables by identifier through that catalog).

Dumping my thoughts here and feedbacks are welcome!

### Option A: Build Catalog inside the ProviderFactory using `OPTIONS`
`IcebergTableProviderFactory` parses `OPTIONS` and uses a `CatalogBuilder` to construct the `Catalog` internally, then creates a normal `IcebergTableProvider`
```sql
CREATE EXTERNAL TABLE my_table
STORED AS ICEBERG
LOCATION 'ignored_or_optional' // this will be ignored if a catalog is configured
OPTIONS (
'datafusion.iceberg.catalog.type' = 'rest', // if catalog type is not configured, it should fall back to create static table
'datafusion.iceberg.catalog.uri' = 'http://localhost:8181',
'datafusion.iceberg.catalog.warehouse' = 's3://bucket/warehouse'
);
```

### Option B: Allow injecting a pre-built Catalog into the factory
Essentially we have
```rust
pub struct IcebergTableProviderFactory {
catalog: Option>, // when it's none, fall back to static table
}
...
IcebergTableProviderFactory::new_with_catalog(Arc)
```

I prefer this as it is much more straight-forward, but one drawback I can think of is users cannot easily use multiple catalogs at the same time. A workaround would look like this:
```rust
state
.table_factories_mut()
.insert("ICEBERG_REST_A".to_string(), Arc::new(IcebergTableProviderFactory(rest_catalog_a)));

state
.table_factories_mut()
.insert("ICEBERG_REST_B".to_string(), Arc::new(IcebergTableProviderFactory(rest_catalog_b)));
```

and then when creating the table using sql:
```sql
CREATE EXTERNAL TABLE my_table
STORED AS ICEBERG_REST_A
...

```

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start at IcebergTableProviderFactory and the DataFusion table_factories_mut registration path, then compare how the existing metadata-JSON LOCATION flow creates static tables. Evaluate the proposed CatalogBuilder and injected-Catalog options; done means CREATE EXTERNAL TABLE can resolve catalog-backed identifiers while preserving the existing static-table behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.