apache / apache/iceberg-rust

REST catalog: client support for atomic multi-table commits (POST /v1/{prefix}/transactions/commit)

Open
#2,784 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

The REST catalog spec defines `POST /v1/{prefix}/transactions/commit`, which applies changes to multiple tables in one atomic operation, and `CommitTableRequest` in this crate already notes that `identifier` "must be present for CommitTransactionRequest" — but there is currently no way to drive that endpoint from iceberg-rust.

Proposal:

- `Transaction::prepare_commit()` — run a transaction's actions and return the resulting `TableCommit` without sending it to a catalog (the per-table building block).
- `RestCatalog::commit_transaction(Vec)` — submit the group as one `CommitTransactionRequest`; the catalog applies all of the changes or none of them.

Kept on `RestCatalog` rather than the `Catalog` trait because multi-table atomicity is a catalog capability rather than a universal guarantee; whether the trait should expose a capability for it seems like a good follow-up discussion.

Use case: a CDC producer (Postgres → Iceberg) that commits row deltas to many tables plus a changelog table per cadence, and needs the group to land atomically so downstream readers never observe a half-applied cut. Validated against Lakekeeper, which implements the endpoint.

I have a working implementation with unit tests and will open a PR.

Contributor guide

Open the contributing guide

Research direction

Start by locating Transaction, RestCatalog, CommitTableRequest, and the existing REST catalog commit tests. Trace how per-table transaction actions are currently prepared and how REST commits are submitted. Done means the proposed prepare_commit and commit_transaction flows are covered by unit tests and submit one atomic CommitTransactionRequest to POST /v1/{prefix}/transactions/commit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.