apache / apache/fluss

[Server] Add the server-side primary-key table bulk-load protocol

Open
#4,224 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Add a complete server-side transaction protocol for atomically loading an empty
primary-key table or partition from externally built KV snapshots.

The server must own the transaction lifecycle, prevent access while the target
is being loaded, validate the submitted manifest, install every bucket, and
publish the target only after the complete result is ready.

The protocol should be usable through the internal Coordinator RPC interface.
A high-level client SDK is tracked separately.

#### Scope

- Add Begin, GetInProgress, Commit, and Abort RPCs.
- Support both a complete primary-key table and one physical partition as the
transaction target.
- Reject non-primary-key and non-empty targets.
- Freeze the target identity, schema, bucket count, remote data directory, and
snapshot IDs when beginning a transaction.
- Fence the target and prevent ordinary access until it becomes active again.
- Allow only one active bulk-load transaction for a target.
- Validate the manifest and require complete coverage of every target bucket.
- Persist the Commit decision before publishing the imported snapshots.
- Restore the imported snapshots on the assigned replicas.
- Publish the complete target atomically after all required replicas are ready.
- Abort transactions that have not crossed the durable Commit decision.
- Resume transactions after Coordinator failover.
- Reconcile TabletServer restart, leader change, and assignment change during
protocol execution.
- Retain terminal results for recovery and remove expired transaction metadata.
- Protect active bulk-load files from orphan cleanup.
- Reject unsupported rolling-upgrade combinations through server capability
checks.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin by locating the internal Coordinator RPC interface and the existing transaction and recovery paths. Trace target fencing, manifest validation, replica restoration, failover and restart reconciliation, and durable commit handling; done means the listed lifecycle operations work for tables or partitions without exposing partial results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.