Support staged table creation transactions
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
Catalog::create_table eagerly creates and registers an empty table. Callers that need Spark-style staged creation cannot use transaction actions to write data before the table becomes visible, then atomically publish the table and its initial snapshot.
The existing Transaction type also assumes that a table already exists: it reloads a current base table and commits updates against its metadata. Table creation has no existing base and must instead fail atomically if another writer creates the table first.
### Describe the solution you would like
Add an opt-in transactional catalog capability for staged table creation.
- Add create_table_transaction to return a transaction backed by catalog-staged table metadata.
- Expose the staged Table so normal transaction actions can write against its schema, partition spec, location, and FileIO.
- Keep the table invisible until the final transaction commit.
- Commit initial metadata updates with an assert-create requirement.
- Do not retry a failed create, because a conflicting table must not be replaced or rebased.
- Implement the capability for MemoryCatalog first, with additional catalogs handled separately.
The existing eager Catalog::create_table operation should remain unchanged. This staged-create capability can also serve as the foundation for later CREATE OR REPLACE support.
### Willingness to contribute
I can contribute to this feature independently.
Implementation: apache/iceberg-rust#3039
Contributor guide
Research direction
Start with Catalog::create_table and the existing Transaction type, then trace how MemoryCatalog stores table metadata and commits updates. Use implementation issue apache/iceberg-rust#3039 as the entry point; done means staged tables remain invisible until commit, initial metadata commits use assert-create semantics, and conflicting creates fail without retry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100