[Umbrella] Support production-ready bulk loading for primary-key tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Fluss currently lacks an efficient way to load a large initial dataset or
backfill a primary-key table. Writing records through the ordinary online
write path is expensive for bounded batch workloads.
This capability is also a foundation for promoting lake-resident data into
Fluss's real-time serving path. Typical use cases include upgrading a lake
table into a real-time table and promoting a cold lake partition into a
real-time partition, without replaying the complete dataset through the
ordinary online write path.
The target capability should build ordinary KV snapshots outside TabletServers,
install them through a server-managed transaction, and make the complete target
visible atomically.
Production use also requires scoped temporary write authorization. A workflow
that succeeds only when clients have unrestricted storage credentials is not
sufficient for production batch insert or lake-to-real-time promotion.
### Solution
#### Scope
The initial implementation supports:
- Empty primary-key tables and partitions.
- Offline construction of ordinary Fluss KV snapshots.
- Server-managed Begin, Commit, and Abort transactions.
- Target fencing while a bulk load is in progress.
- Atomic publication after every bucket and replica is ready.
- Coordinator failover and assignment changes during transaction processing.
- A public client SDK for building and committing bulk-load files.
- Scoped temporary write authorization for build sessions.
- Flink batch insert as the first engine integration.
#### Sub Tasks
- [ ] #4223
- [ ] #4224
- [ ] #4225
- [ ] #4226
- [ ] #4227
The tasks are expected to be delivered in the listed order.
### Anything else?
### Completion criteria
This umbrella issue is complete when:
- An externally built KV snapshot can be validated, adopted, and restored by Fluss.
- The server provides a complete and recoverable Begin/Commit/Abort lifecycle.
- The client SDK exposes the complete bulk-load workflow.
- Build sessions can obtain scoped temporary write authorization without
serializing credentials into build contexts or engine checkpoints.
- Flink batch insert can load an empty primary-key table or partition and expose
the result atomically.
- Failure, abort, Coordinator failover, TabletServer restart, and assignment
change paths have appropriate regression coverage.
- Compatibility behavior and user documentation are complete.
#### Non-goals
- Streaming bulk load.
- Loading into a non-empty target.
- Non-primary-key tables.
- Atomic commit across multiple independent targets.
- Spark integration.
- A complete general-purpose authentication or authorization system.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing subtasks #4223–#4227 in their listed order, then compare the proposed workflow with the scope and completion criteria here. The work is done when externally built KV snapshots, the recoverable Begin/Commit/Abort lifecycle, client SDK, scoped authorization, Flink batch insert, failure coverage, compatibility behavior, and documentation are complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100