apache / apache/polaris

Catalog-level branching for atomic multi-table operations

Open
#3,469 16 comments 9 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Is your feature request related to a problem? Please describe.

Polaris supports per-table branching via Iceberg's branch refs, but operating on branches across many tables is manual and non-atomic. We have 100s of tables. The pain points:

- No cross-table branch operations: Creating/merging a branch for N tables requires N individual API calls
- No atomic multi-table promotion: Merging tables one-by-one means clients can observe inconsistent states across tables mid-promotion.
- Slow branch listing at scale: Discovering which tables have a given branch requires loading each table's metadata individually

### Describe the solution you'd like

Catalog-level branching that enables:

- Creating/merging a branch across a namespace or set of tables in a single operation
- Atomic promotion so all tables flip together
- Efficient branch listing without loading each table's metadata

Conceptually:
```
polaris branch create backfill-2024 --from main --namespace my_ns
# ... run operations against branch ...
polaris branch merge backfill-2024 --into main # All tables promoted atomically
```

### Describe alternatives you've considered

**Nessie**: Provides catalog-level versioning with Git-like semantics, atomic multi-table commits, and efficient branch operations—but requires adopting a separate catalog
**Scripted per-table branching**: What we do today—operationally complex, non-atomic, doesn't scale well
**Blue-green table swaps**: Duplicate tables, swap via rename—error-prone at 200 tables

### Additional context

Nessie has proven this pattern works well. Given discussions around Polaris + Nessie convergence, this seems like a natural feature to bring into Polaris. Happy to contribute to design/implementation.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by mapping Polaris's catalog and table-branch APIs, then define the scope and atomicity requirements for namespace or multi-table create, merge, and branch listing; done means an agreed design and implementation path for these operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.