apache / apache/paimon

[Feature] PIP-30: Introduce coordinator-based commit for Flink sink

Open
#8,220 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

This is the umbrella issue for [PIP-30](https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink) — moving Paimon's Flink committer from a downstream `CommitterOperator` to a JobManager-side `OperatorCoordinator`. The target is Flink jobs writing to unaware-bucket append tables, where this enables region failover and removes the commit-side fan-in shuffle for large-parallelism Flink-to-Paimon write jobs.

Recent design discussion (with updated diagrams that didn't render cleanly on cwiki):
- Mailing-list thread: https://lists.apache.org/thread/5n1mwc7wc3cfjogz56w9k5ql3zyl00lw
- Updated design doc: https://docs.google.com/document/d/1asWKzoytfeB1D8bS_yRIAHnpR40frLf0fnPn2-WSL74/edit

### Solution

The work will be split across a series of PRs to keep each one independently reviewable. Tentative breakdown (subject to change as reviews progress):

- [x] Decouple `Committer.Context` from operator-only assumptions.
- [x] Introduce `WriteOperatorCoordinator` as the JobManager-side commit driver. And wire the unaware-bucket append table to coordinator-based commit.
- [ ] Follow-ups as needed.
- [ ] End-of-input handling
- [x] Idle watermark handling
- [ ] Add failover IT case or extend existing IT case to cover coordinator commit
- [ ] Auto-tag-for-savepoint on the coordinator path
- [ ] Concurrent checkpoints support
- [ ] Document

Links to per-PR issues will be added here as they are opened.
- https://github.com/apache/paimon/pull/8221
- https://github.com/apache/paimon/pull/8401

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the PIP-30 design document and the linked PRs #8221 and #8401 to understand the coordinator-based commit work already underway. Review the remaining unchecked items in this umbrella issue, including end-of-input handling, failover IT coverage, savepoint tagging, concurrent checkpoints, and documentation; done criteria depend on the follow-up selected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.