apache / apache/doris

[Feature](paimon) Support Branch/Tag lifecycle and branch writes

Open
#67,336 0 comments 0 reactions 0 assignees View on GitHub
area/catalog kind/feature tracking issue
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Background

Apache Doris can read Apache Paimon branches and tags, including branch-specific schemas and retained tag snapshots. However, Doris cannot currently create or drop Paimon branches/tags, and Paimon branches cannot be used as Doris write targets.

This issue tracks the Branch/Tag lifecycle and branch-write follow-up to #65086.

Upstream reference: https://paimon.apache.org/docs/1.4/maintenance/manage-branches/

## Current Status

- Read from `table@branch(name)`: supported.
- Read from `table@tag(name)`: supported.
- Query `$branches` and `$tags` system tables: supported.
- Create/drop Branch or Tag from Doris SQL: not supported.
- Write to a Branch from Doris SQL: not supported.
- Tag writes: must remain unsupported because tags are immutable references.
- Fast Forward / Merge / Replace operations: not supported from Doris.

## Goals

### Branch and Tag Lifecycle

- Create a branch from the current table state, a snapshot, or a tag.
- Drop and rename branches.
- Create a tag from the latest or a specified snapshot.
- Support tag retention when the Paimon version and catalog provide it.
- Drop and rename tags.
- Support `IF EXISTS`, `IF NOT EXISTS`, and idempotent retry semantics where applicable.
- Invalidate Doris metadata, schema, snapshot, and partition caches after reference changes.

The canonical Doris interface should reuse the existing external-table Branch/Tag DDL model. Procedure aliases may be added by the separate maintenance-procedure issue, but must call the same underlying implementation.

### Branch Writes

- Support `INSERT INTO` and `INSERT OVERWRITE` on a Paimon branch.
- Support row-level `UPDATE`, `DELETE`, and `MERGE` on a Paimon branch where the target table supports them.
- Bind analysis, distribution, writer initialization, commit, abort, and reconciliation to the selected branch table and its schema generation.
- Keep transaction identity, metadata refresh, and cache keys scoped to the branch.
- Reject every write to a tag during analysis.
- Never fall back to `main` when branch resolution or branch commit fails.

### Promotion and Conflict Semantics

- Support Fast Forward from a branch to `main`.
- Define Merge / Replace behavior according to the Paimon version used by Doris.
- Detect incompatible branch ancestry, schema history, and concurrent changes before promotion.
- Define behavior when `main` or the branch advances during a Doris statement.
- Preserve Paimon commit/abort and retry semantics on branch-targeted transactions.

### Interoperability

- Read and write branches created by Flink or Spark.
- Make Doris-created branches and tags visible and usable from Flink and Spark.
- Cover filesystem, HMS, and REST catalogs.

## Acceptance Criteria

- Branch `INSERT`, overwrite, and row-level DML mutate only the selected branch.
- Main, other branches, and tags remain unchanged after a successful or failed branch write.
- Tag writes fail during analysis with a clear error.
- Branch schema evolution is resolved from the branch rather than the latest `main` schema.
- Fast Forward succeeds only for a valid ancestry and leaves Doris caches consistent with Paimon metadata.
- Commit timeout, retry, abort, FE-side reconciliation, and concurrent promotion cases have regression coverage.
- Cross-engine tests cover branches/tags created and consumed by Doris, Spark, and Flink.

## Suggested Delivery Phases

1. Branch/Tag create, drop, read interoperability, and immutable Tag enforcement.
2. Branch `INSERT INTO` / `INSERT OVERWRITE` with transaction isolation.
3. Branch `UPDATE` / `DELETE` / `MERGE` and Fast Forward.
4. Merge / Replace, retention, rename, and extended concurrency coverage.

## Non-goals

- A generic Catalog procedure framework.
- Snapshot expiration, orphan-file cleanup, compaction scheduling, or rollback procedures.
- Rust FFI write backend work.

## Related

- Paimon write tracking: #65086
- Paimon maintenance procedures and actions: #67337

Contributor guide

Open the contributing guide

Research direction

No Doris files or tests are named. Start with related issue #65086 and the linked Paimon branch-management reference, then use the listed delivery phases to scope the work. Done requires branch lifecycle support, isolated branch writes, immutable-tag enforcement, promotion semantics, cache consistency, and the specified regression and cross-engine coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.