apache / apache/fluss

[Client] Add the primary-key table bulk-load client SDK

Open
#4,225 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Provide a public client SDK over the server-side bulk-load protocol.

The SDK should hide RPC messages, manifest assembly, snapshot file handling,
and Commit retry details while supporting both single-process builders and
distributed execution engines.

### Scope

- Expose BulkLoad through the Fluss connection.
- Add a client for Begin, GetInProgress, Commit, and Abort.
- Return an immutable and serializable build context from Begin.
- Add `openBuildSession` as the entry point for local build resources.
- Create bucket writers through the build session.
- Route and validate rows against the frozen target metadata.
- Deduplicate primary-key rows locally, with the last row for a key winning.
- Build ordinary Fluss KV snapshot files for each bucket.
- Support an explicit log end offset when required by the caller.
- Assemble and publish the complete manifest internally.
- Retry Commit within a caller-provided await budget when the outcome is
retriable or unknown.
- Close all writers still owned by a build session when the session is closed.
- Keep credentials and process-local resources out of the serializable build
context.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Fluss connection and the server-side bulk-load protocol, then trace the Begin, GetInProgress, Commit, and Abort entry points. Review how build sessions, bucket writers, snapshot files, manifests, and commit retries are represented; done means the public SDK covers the listed local and distributed workflows while keeping the build context serializable and resources properly closed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.