apache / apache/iceberg-rust

Support staged table creation transactions

Open
#3,040 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Catalog::create_table eagerly creates and registers an empty table. Callers that need Spark-style staged creation cannot use transaction actions to write data before the table becomes visible, then atomically publish the table and its initial snapshot.

The existing Transaction type also assumes that a table already exists: it reloads a current base table and commits updates against its metadata. Table creation has no existing base and must instead fail atomically if another writer creates the table first.

### Describe the solution you would like

Add an opt-in transactional catalog capability for staged table creation.

- Add create_table_transaction to return a transaction backed by catalog-staged table metadata.
- Expose the staged Table so normal transaction actions can write against its schema, partition spec, location, and FileIO.
- Keep the table invisible until the final transaction commit.
- Commit initial metadata updates with an assert-create requirement.
- Do not retry a failed create, because a conflicting table must not be replaced or rebased.
- Implement the capability for MemoryCatalog first, with additional catalogs handled separately.

The existing eager Catalog::create_table operation should remain unchanged. This staged-create capability can also serve as the foundation for later CREATE OR REPLACE support.

### Willingness to contribute

I can contribute to this feature independently.

Implementation: apache/iceberg-rust#3039

Contributor guide

Open the contributing guide

Research direction

Start with Catalog::create_table and the existing Transaction type, then trace how MemoryCatalog stores table metadata and commits updates. Use implementation issue apache/iceberg-rust#3039 as the entry point; done means staged tables remain invisible until commit, initial metadata commits use assert-create semantics, and conflicting creates fail without retry.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.