[Feature] PIP-30: Introduce coordinator-based commit for Flink sink
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
This is the umbrella issue for [PIP-30](https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink) — moving Paimon's Flink committer from a downstream `CommitterOperator` to a JobManager-side `OperatorCoordinator`. The target is Flink jobs writing to unaware-bucket append tables, where this enables region failover and removes the commit-side fan-in shuffle for large-parallelism Flink-to-Paimon write jobs.
Recent design discussion (with updated diagrams that didn't render cleanly on cwiki):
- Mailing-list thread: https://lists.apache.org/thread/5n1mwc7wc3cfjogz56w9k5ql3zyl00lw
- Updated design doc: https://docs.google.com/document/d/1asWKzoytfeB1D8bS_yRIAHnpR40frLf0fnPn2-WSL74/edit
### Solution
The work will be split across a series of PRs to keep each one independently reviewable. Tentative breakdown (subject to change as reviews progress):
- [x] Decouple `Committer.Context` from operator-only assumptions.
- [x] Introduce `WriteOperatorCoordinator` as the JobManager-side commit driver. And wire the unaware-bucket append table to coordinator-based commit.
- [ ] Follow-ups as needed.
- [ ] End-of-input handling
- [x] Idle watermark handling
- [ ] Add failover IT case or extend existing IT case to cover coordinator commit
- [ ] Auto-tag-for-savepoint on the coordinator path
- [ ] Concurrent checkpoints support
- [ ] Document
Links to per-PR issues will be added here as they are opened.
- https://github.com/apache/paimon/pull/8221
- https://github.com/apache/paimon/pull/8401
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the PIP-30 design document and the linked PRs #8221 and #8401 to understand the coordinator-based commit work already underway. Review the remaining unchecked items in this umbrella issue, including end-of-input handling, failover IT coverage, savepoint tagging, concurrent checkpoints, and documentation; done criteria depend on the follow-up selected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100