[Server] Add the server-side primary-key table bulk-load protocol
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Add a complete server-side transaction protocol for atomically loading an empty
primary-key table or partition from externally built KV snapshots.
The server must own the transaction lifecycle, prevent access while the target
is being loaded, validate the submitted manifest, install every bucket, and
publish the target only after the complete result is ready.
The protocol should be usable through the internal Coordinator RPC interface.
A high-level client SDK is tracked separately.
#### Scope
- Add Begin, GetInProgress, Commit, and Abort RPCs.
- Support both a complete primary-key table and one physical partition as the
transaction target.
- Reject non-primary-key and non-empty targets.
- Freeze the target identity, schema, bucket count, remote data directory, and
snapshot IDs when beginning a transaction.
- Fence the target and prevent ordinary access until it becomes active again.
- Allow only one active bulk-load transaction for a target.
- Validate the manifest and require complete coverage of every target bucket.
- Persist the Commit decision before publishing the imported snapshots.
- Restore the imported snapshots on the assigned replicas.
- Publish the complete target atomically after all required replicas are ready.
- Abort transactions that have not crossed the durable Commit decision.
- Resume transactions after Coordinator failover.
- Reconcile TabletServer restart, leader change, and assignment change during
protocol execution.
- Retain terminal results for recovery and remove expired transaction metadata.
- Protect active bulk-load files from orphan cleanup.
- Reject unsupported rolling-upgrade combinations through server capability
checks.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin by locating the internal Coordinator RPC interface and the existing transaction and recovery paths. Trace target fencing, manifest validation, replica restoration, failover and restart reconciliation, and durable commit handling; done means the listed lifecycle operations work for tables or partitions without exposing partial results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100