[Feature] Spark WAP (write, audit, publish) branch support
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
## Problem
When combining batch generated & streaming data, a single large snapshot in a Paimon table can stall Flink checkpointing for a long time.
```mermaid
flowchart TD
Spark --> BatchData[(Batch Data)]
BatchData --> FlinkJoin
EventData[(Event Data)] --> FlinkJoin
FlinkJoin --> JoinedData[(Joined Data)]
```
For example, if we rewrite every row in `Batch Data` in a single snapshot from Spark, the realtime `Event Data` stalls until the full table is scanned.
## Solution
Using Paimon's branching support, we can split the Spark changes up into smaller commits on a staging branch, then fast-forward the main branch at the end of the batch.
This allows us to have smaller, incremental snapshots while keeping the batch update atomic & avoiding partial data if the batch fails half way through.
### Solution
This is achievable today, but it would be great to have a similar feature to Iceberg, where you can specify `write.wap.enabled` and the branch is created & merged automatically.
Prior art from Iceberg: https://iceberg.apache.org/docs/latest/branching/#audit-branch
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Paimon's existing branching support through its Spark integration, then compare the requested behavior with Iceberg's audit branch documentation. Define how write.wap.enabled creates and merges the branch, and verify that batch updates remain atomic while producing smaller incremental snapshots without stalling streaming data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100