[Flink] Support Flink batch insert into Fluss primary-key tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Integrate the BulkLoad client SDK with the Flink connector so a bounded
`INSERT INTO` job can efficiently populate an empty Fluss primary-key table or
partition.
The Flink integration should coordinate one server transaction, build buckets
in parallel, and publish the target only after every bucket has completed.
## Scope
- Add an explicit connector option for selecting the BulkLoad path.
- Enable BulkLoad only for bounded Flink jobs and supported primary-key tables.
- Begin one BulkLoad transaction for the physical target.
- Distribute the immutable build context to parallel build tasks.
- Assign every input row to its Fluss bucket.
- Build bucket snapshots in parallel through `openBuildSession`.
- Collect one completed result for every target bucket.
- Commit the complete manifest through one coordinated commit path.
- Abort or report a clear failure when the job cannot complete before the
durable Commit decision.
- Keep temporary credentials out of Flink records, committables, checkpoints,
and savepoints.
- Document configuration, supported cases, failure behavior, and operational
limitations.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the Flink connector's bounded INSERT handling and the BulkLoad client SDK entry points, then trace how transactions, build tasks, bucket results, and commits are coordinated. Done means supported bounded jobs can bulk-load primary-key targets with one coordinated commit, clear failure handling, protected credentials, and documented configuration and limitations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100