Catalog-level branching for atomic multi-table operations
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Is your feature request related to a problem? Please describe.
Polaris supports per-table branching via Iceberg's branch refs, but operating on branches across many tables is manual and non-atomic. We have 100s of tables. The pain points:
- No cross-table branch operations: Creating/merging a branch for N tables requires N individual API calls
- No atomic multi-table promotion: Merging tables one-by-one means clients can observe inconsistent states across tables mid-promotion.
- Slow branch listing at scale: Discovering which tables have a given branch requires loading each table's metadata individually
### Describe the solution you'd like
Catalog-level branching that enables:
- Creating/merging a branch across a namespace or set of tables in a single operation
- Atomic promotion so all tables flip together
- Efficient branch listing without loading each table's metadata
Conceptually:
```
polaris branch create backfill-2024 --from main --namespace my_ns
# ... run operations against branch ...
polaris branch merge backfill-2024 --into main # All tables promoted atomically
```
### Describe alternatives you've considered
**Nessie**: Provides catalog-level versioning with Git-like semantics, atomic multi-table commits, and efficient branch operations—but requires adopting a separate catalog
**Scripted per-table branching**: What we do today—operationally complex, non-atomic, doesn't scale well
**Blue-green table swaps**: Duplicate tables, swap via rename—error-prone at 200 tables
### Additional context
Nessie has proven this pattern works well. Given discussions around Polaris + Nessie convergence, this seems like a natural feature to bring into Polaris. Happy to contribute to design/implementation.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by mapping Polaris's catalog and table-branch APIs, then define the scope and atomicity requirements for namespace or multi-table create, merge, and branch listing; done means an agreed design and implementation path for these operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100