apache / apache/iceberg

Add branch support for `RewriteManifests` operation

Open
#15,981 2 comments 10 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

### What

Add support for committing a `rewriteManifests` operation to a named branch via `toBranch(String branch)`.

### Why

**Manifest clustering for scan performance**

In tables with many partitions, manifest files that span multiple partitions cannot be pruned during planning even after the manifest list has been read. Rewriting manifests clustered by partition (`.clusterBy(DataFile::partition)`) significantly reduces the metadata read overhead for single-partition scans.

**Write-Audit-Publish (WAP) workflow**

Our ingestion pipeline follows a WAP pattern: data is written to a branch, verified, and then merged into `main`. Before merging, we want to compact and cluster the branch's manifests so that `main` remains efficient after the merge. Without branch support in `rewriteManifests`, this step is not possible without merging unoptimised metadata first.

### Current behaviour

`RewriteManifests` extends `SnapshotUpdate`, which provides a default `toBranch` implementation that always throws:

```java
default ThisT toBranch(String branch) {
throw new UnsupportedOperationException(
String.format(
"Cannot commit to branch %s: %s does not support branch commits",
branch, this.getClass().getName()));
}
```

`BaseRewriteManifests` does not override this method, so calling `.toBranch(...)` throws `UnsupportedOperationException` at runtime.

There is a second, related issue: the `apply` method in `BaseRewriteManifests` reads manifests from `base.currentSnapshot()` (always the `main` snapshot) rather than from the snapshot parameter passed in by `SnapshotProducer`, which represents the tip of the target branch. Even if `toBranch` were unblocked, the rewrite would operate on the wrong snapshot.

### Expected behaviour

- `table.rewriteManifests().toBranch("branch").commit()` rewrites the manifests of the named branch without affecting `main`.
- If the branch does not yet exist, it is created (consistent with the behaviour of other `SnapshotUpdate` operations such as `AppendFiles`).
- Passing `null` or a tag name as the branch argument throws `IllegalArgumentException` (same validation as the rest of the API).

### Query engine

Other

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with BaseRewriteManifests.apply, SnapshotUpdate.toBranch, and SnapshotProducer to trace how the target snapshot is selected and how branch commits are validated. Compare branch handling with AppendFiles and verify rewriteManifests().toBranch("branch").commit() creates or updates the branch without changing main; also cover null and tag-name validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.