apache / apache/hudi

[FEATURE] Branches and tags (timeline branching) for safe rollouts, Hudi upgrades, and reproducible reads

Open
#19,066 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
First-class, named branches and tags for a Hudi table (timeline branching).

- A branch is a named, movable, writable reference to an independent line of commits with its own timeline. A writer can fork a branch off main, commit to it, and have those commits be completely invisible to every reader and writer on main until the branch is explicitly published (fast-forwarded) into main. Publishing flips the table's current view atomically, so readers move to the new data with no partial state.
- A tag is a named, immutable reference to a single instant — a stable, reproducible, cleaning-protected snapshot of the table at a point in time. Tags have no timeline of their own; they just label an instant that already exists on a branch's timeline.

The special main branch is the default and is backed by today's timeline, so existing tables behave exactly as before. Branches and tags compose: tag a branch's head to freeze a validated state, or fork a branch from a tag to start a new line of work from a frozen point.

**Why this feature is needed:**
Hudi today exposes a single, linear timeline: the moment a commit transitions to COMPLETED, it is visible to every reader. There is no supported way to (a) stage commits "off to the side," validate them, and atomically expose them, or (b) keep a stable, named, immutable snapshot for reproducibility. The only remedies for a bad batch are reactive — restore/rollback — which are destructive and global (they rewrite the one timeline for everyone) and run only after bad data has already been read. And as.of.instant time-travel only works if the underlying file slices were not already cleaned — there is no named, cleaning-protected, catalog-visible snapshot that survives and can be shared by name.

This is especially painful for rollouts and Hudi version upgrades. Today you cannot run a new/upgraded ingestion writer against a live production table and validate its output before exposing it, without standing up a parallel table and re-pointing every consumer by hand. A regression introduced by a new Hudi version or a pipeline change is visible to consumers immediately.

Branches + tags enable error-free, seamless upgrades and rollouts:
- Tag the last-known-good production state so it can be reproduced/compared against, and rolled back to, at any time.
- The existing (current-version) writer keeps serving production on main.
- A new writer — running the upgraded Hudi version or new logic — forks a branch and writes there, fully isolated.
- After a few commits, the branch's data is validated independently of production (and can be compared against the pre-upgrade tag).
- Once everything looks good, the change is promoted to the main ingestion writer (publish the branch / cut the main writer over), and the published snapshot can be tagged for audit.
- If validation fails, the branch is simply dropped — production on main never saw the change.

This gives blast-radius-free testing of upgrades on real, current production data (branches), plus a permanent, shareable contract for "exactly these rows" (tags) — instead of relying on sampled/stale/copied tables or best-effort time travel.

### User Experience

**How users will use this feature:**
- Configuration changes needed
- hoodie.table.branching.enabled: Table property to opt into branches/tags (additive; main-only tables produce no new files).
- hoodie.write.branch: Branch this writer commits to.
- hoodie.read.branch: Branch to read (snapshot/incremental).
- hoodie.read.tag: Tag to read.
- API changes
- Spark SQL DDL — branches: ALTER TABLE t CREATE BRANCH [AS OF ...], ALTER TABLE t DROP BRANCH [IF EXISTS] , SELECT *
FROM t VERSION AS OF BRANCH ''.
- Spark SQL DDL — tags: ALTER TABLE t CREATE TAG [AS OF ...] [RETAIN MONTHS], ALTER TABLE t DROP TAG [IF EXISTS] ,
SELECT * FROM t VERSION AS OF TAG ''.
- Procedures: CALL publish_branch(table => 't', branch => '') (fast-forward main to a branch head); CALL create_branch(...),
CALL drop_branch(...), CALL create_tag(...), CALL drop_tag(...), CALL list_refs(table => 't').
- DataSource options: hoodie.write.branch, hoodie.read.branch, hoodie.read.tag.
- Hudi Streamer / Flink: --hoodie-conf hoodie.write.branch= / 'hoodie.write.branch' = ''.

- Usage examples
- ALTER TABLE events CREATE TAG pre_upgrade_1_1_0; -- Freeze the last-known-good production state before touching anything.
- ALTER TABLE events CREATE BRANCH upgrade_1_1_0; -- Stand up an upgrade/test branch forked from the current production view.
- SELECT count(*), sum(amount) FROM events BRANCH AS OF 'upgrade_1_1_0';
- SELECT count(*), sum(amount) FROM events TAG AS OF 'pre_upgrade_1_1_0';
- CALL publish_branch(table => 'events', branch => 'upgrade_1_1_0'); -- This switches upgrade_1_1_0 as main branch.
- ALTER TABLE events DROP BRANCH upgrade_1_1_0;
- df.write.format("hudi")
.option("hoodie.write.branch", "upgrade_1_1_0")
.option(/* usual Hudi write options */)
.mode(Append).save(basePath) -- // New/upgraded writer targets the branch
- spark.read.format("hudi").option("hoodie.read.branch", "upgrade_1_1_0").load(basePath)
- spark.read.format("hudi").option("hoodie.read.tag", "pre_upgrade_1_1_0").load(basePath)

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No): Yes, adds Spark SQL DDL (CREATE/DROP BRANCH, CREATE/DROP TAG, BRANCH/TAG AS OF 'XXXXX'), publish_branch / create_tag / drop_tag / list_refs procedures, new DataSource/Streamer/Flink write & read options, and a new HMS published-pointer contract for cross-engine resolution.
- Does this change storage format? (Yes/No): Yes, introduces a per-branch timeline layout under .hoodie/branches//timeline/ and a ref registry under .hoodie/refs/{branches,tags}/, gated behind a table-version bump and hoodie.table.branching.enabled. Data files remain shared across refs (addressed by instant time; no copy). Tables that never create a branch or tag are byte-for-byte unchanged.
- Justification: The change affects timeline storage, reader/writer resolution, cross-engine catalog (HMS) behavior, and retention — the cleaner/archival must treat every live branch and tag head as a GC root (tags generalize the existing savepoint-protection mechanism). This cross-cutting, backward-compatibility-sensitive scope warrants a design review via an RFC.

Contributor guide

No contributing guide indexed for this repository

Research direction

No implementation files, tests, or entry points are named. Start by turning the proposed RFC into a design covering timeline storage, reader and writer resolution, HMS published-pointer behavior, and cleaner/archival retention. Done means branches and tags satisfy the stated isolation, atomic publishing, reproducibility, and backward-compatibility requirements across the listed APIs and engines.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.