[Feature] Support bucket commit in paimon
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Currently `Paimon` will create a global commit operator for flink job which makes all operators in the job as one region. When a task failed, it causes global failover. I'd like to support bucket commit in paimon, then we can build commit operator for each sink and support region failover in flink jobs
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing global commit operator in Paimon's Flink integration and how sink operators participate in job failover. Define how commit operators should be scoped per bucket or sink, then verify that a task failure can trigger regional rather than global failover. Done means bucket commit support is available without making all job operators one region.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100