apache / apache/paimon

[Feature] Spark WAP (write, audit, publish) branch support

Open
#9,690 0 comments 7 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

## Problem
When combining batch generated & streaming data, a single large snapshot in a Paimon table can stall Flink checkpointing for a long time.

```mermaid
flowchart TD
Spark --> BatchData[(Batch Data)]
BatchData --> FlinkJoin
EventData[(Event Data)] --> FlinkJoin
FlinkJoin --> JoinedData[(Joined Data)]
```

For example, if we rewrite every row in `Batch Data` in a single snapshot from Spark, the realtime `Event Data` stalls until the full table is scanned.

## Solution
Using Paimon's branching support, we can split the Spark changes up into smaller commits on a staging branch, then fast-forward the main branch at the end of the batch.
This allows us to have smaller, incremental snapshots while keeping the batch update atomic & avoiding partial data if the batch fails half way through.

### Solution

This is achievable today, but it would be great to have a similar feature to Iceberg, where you can specify `write.wap.enabled` and the branch is created & merged automatically.

Prior art from Iceberg: https://iceberg.apache.org/docs/latest/branching/#audit-branch

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Paimon's existing branching support through its Spark integration, then compare the requested behavior with Iceberg's audit branch documentation. Define how write.wap.enabled creates and merges the branch, and verify that batch updates remain atomic while producing smaller incremental snapshots without stalling streaming data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.