datafusion-contrib / datafusion-contrib/StreamFusion
Fluss source: consume Java-client Arrow batches
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Goal
Add a StreamFusion Fluss source path that keeps Arrow buffers returned by the Fluss Java client instead of first materializing live-log records as Flink RowData.
This should use Fluss Java APIs for split/offset/checkpoint coordination and direct Arrow polling for the data plane. It should not depend on LogScannerImpl, other @Internal classes, or unpublished locally built Fluss artifacts.
Upstream dependency
Wait until the direct Arrow polling API is public and available in a published Fluss artifact:
- Apache Fluss issue: https://github.com/apache/fluss/issues/4005
- Apache Fluss PR: https://github.com/apache/fluss/pull/4006
The upstream PR exposes LogScanner.pollRecordBatch, preserves the stored PK-table changelog sidecar, and supports non-empty top-level Arrow projections. Fluss 0.9.1 does not contain the batch API; the current 1.0 snapshot exposes only an internal append-only, unprojected path and is not an acceptable production dependency.
Initial admission whitelist
Accelerate only when all applicable contracts have been verified:
table.log.format = ARROW.- The live/unbounded log phase can return owned Arrow batches directly.
- Append-only tables, plus primary-key changelogs only when the public API includes aligned per-row change types.
- Full scans or verified non-empty top-level projections.
- Supported Arrow/Flink types and schema evolution.
- Supported client security and configuration options.
- Flink-equivalent split assignment, startup offsets, partition discovery, checkpoint recovery, watermarks, idleness, and metrics.
- Explicit Arrow ownership and StreamFusion off-heap accounting.
Everything outside the whitelist must fall back at planning time to the stock Flink Fluss source. Snapshot/bounded reads may remain on the ordinary Flink row path until Fluss exposes an equivalent verified Arrow snapshot contract.
Acceptance criteria
- Depend on a released Fluss artifact containing the public Arrow polling API.
- Add a planner matcher that implements the whitelist and explains every fallback.
- Preserve append-only and PK changelog semantics, offsets, checkpoint recovery, projection ordering, and schema evolution.
- Prove allocator/version compatibility at the Java-to-native Arrow boundary.
- Add integration and failover tests against the stock Flink Fluss connector as the semantic oracle.
- Benchmark the direct Arrow path against the existing row-materializing path.
- Update
docs/connectors/fluss.mdwith the exact accelerated and fallback circumstances.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by checking Apache Fluss issues #4005 and #4006 and confirming that a released artifact provides the public Arrow polling API. Then trace the planner matcher, the stock Flink Fluss connector used as the semantic oracle, and docs/connectors/fluss.md. Done means the whitelist and fallback explanations, integration and failover tests, compatibility checks, benchmark, and documentation updates are complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, documentation, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100