[Feature](paimon) Support Branch/Tag lifecycle and branch writes
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Background
Apache Doris can read Apache Paimon branches and tags, including branch-specific schemas and retained tag snapshots. However, Doris cannot currently create or drop Paimon branches/tags, and Paimon branches cannot be used as Doris write targets.
This issue tracks the Branch/Tag lifecycle and branch-write follow-up to #65086.
Upstream reference: https://paimon.apache.org/docs/1.4/maintenance/manage-branches/
## Current Status
- Read from `table@branch(name)`: supported.
- Read from `table@tag(name)`: supported.
- Query `$branches` and `$tags` system tables: supported.
- Create/drop Branch or Tag from Doris SQL: not supported.
- Write to a Branch from Doris SQL: not supported.
- Tag writes: must remain unsupported because tags are immutable references.
- Fast Forward / Merge / Replace operations: not supported from Doris.
## Goals
### Branch and Tag Lifecycle
- Create a branch from the current table state, a snapshot, or a tag.
- Drop and rename branches.
- Create a tag from the latest or a specified snapshot.
- Support tag retention when the Paimon version and catalog provide it.
- Drop and rename tags.
- Support `IF EXISTS`, `IF NOT EXISTS`, and idempotent retry semantics where applicable.
- Invalidate Doris metadata, schema, snapshot, and partition caches after reference changes.
The canonical Doris interface should reuse the existing external-table Branch/Tag DDL model. Procedure aliases may be added by the separate maintenance-procedure issue, but must call the same underlying implementation.
### Branch Writes
- Support `INSERT INTO` and `INSERT OVERWRITE` on a Paimon branch.
- Support row-level `UPDATE`, `DELETE`, and `MERGE` on a Paimon branch where the target table supports them.
- Bind analysis, distribution, writer initialization, commit, abort, and reconciliation to the selected branch table and its schema generation.
- Keep transaction identity, metadata refresh, and cache keys scoped to the branch.
- Reject every write to a tag during analysis.
- Never fall back to `main` when branch resolution or branch commit fails.
### Promotion and Conflict Semantics
- Support Fast Forward from a branch to `main`.
- Define Merge / Replace behavior according to the Paimon version used by Doris.
- Detect incompatible branch ancestry, schema history, and concurrent changes before promotion.
- Define behavior when `main` or the branch advances during a Doris statement.
- Preserve Paimon commit/abort and retry semantics on branch-targeted transactions.
### Interoperability
- Read and write branches created by Flink or Spark.
- Make Doris-created branches and tags visible and usable from Flink and Spark.
- Cover filesystem, HMS, and REST catalogs.
## Acceptance Criteria
- Branch `INSERT`, overwrite, and row-level DML mutate only the selected branch.
- Main, other branches, and tags remain unchanged after a successful or failed branch write.
- Tag writes fail during analysis with a clear error.
- Branch schema evolution is resolved from the branch rather than the latest `main` schema.
- Fast Forward succeeds only for a valid ancestry and leaves Doris caches consistent with Paimon metadata.
- Commit timeout, retry, abort, FE-side reconciliation, and concurrent promotion cases have regression coverage.
- Cross-engine tests cover branches/tags created and consumed by Doris, Spark, and Flink.
## Suggested Delivery Phases
1. Branch/Tag create, drop, read interoperability, and immutable Tag enforcement.
2. Branch `INSERT INTO` / `INSERT OVERWRITE` with transaction isolation.
3. Branch `UPDATE` / `DELETE` / `MERGE` and Fast Forward.
4. Merge / Replace, retention, rename, and extended concurrency coverage.
## Non-goals
- A generic Catalog procedure framework.
- Snapshot expiration, orphan-file cleanup, compaction scheduling, or rollback procedures.
- Rust FFI write backend work.
## Related
- Paimon write tracking: #65086
- Paimon maintenance procedures and actions: #67337
Contributor guide
Research direction
No Doris files or tests are named. Start with related issue #65086 and the linked Paimon branch-management reference, then use the listed delivery phases to scope the work. Done requires branch lifecycle support, isolated branch writes, immutable-tag enforcement, promotion semantics, cache consistency, and the specified regression and cross-engine coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100