datafusion-contrib / datafusion-contrib/StreamFusion

Fluss source: consume Java-client Arrow batches

Open
#25 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Goal

Add a StreamFusion Fluss source path that keeps Arrow buffers returned by the Fluss Java client instead of first materializing live-log records as Flink RowData.

This should use Fluss Java APIs for split/offset/checkpoint coordination and direct Arrow polling for the data plane. It should not depend on LogScannerImpl, other @Internal classes, or unpublished locally built Fluss artifacts.

Upstream dependency

Wait until the direct Arrow polling API is public and available in a published Fluss artifact:

The upstream PR exposes LogScanner.pollRecordBatch, preserves the stored PK-table changelog sidecar, and supports non-empty top-level Arrow projections. Fluss 0.9.1 does not contain the batch API; the current 1.0 snapshot exposes only an internal append-only, unprojected path and is not an acceptable production dependency.

Initial admission whitelist

Accelerate only when all applicable contracts have been verified:

  • table.log.format = ARROW.
  • The live/unbounded log phase can return owned Arrow batches directly.
  • Append-only tables, plus primary-key changelogs only when the public API includes aligned per-row change types.
  • Full scans or verified non-empty top-level projections.
  • Supported Arrow/Flink types and schema evolution.
  • Supported client security and configuration options.
  • Flink-equivalent split assignment, startup offsets, partition discovery, checkpoint recovery, watermarks, idleness, and metrics.
  • Explicit Arrow ownership and StreamFusion off-heap accounting.

Everything outside the whitelist must fall back at planning time to the stock Flink Fluss source. Snapshot/bounded reads may remain on the ordinary Flink row path until Fluss exposes an equivalent verified Arrow snapshot contract.

Acceptance criteria

  • Depend on a released Fluss artifact containing the public Arrow polling API.
  • Add a planner matcher that implements the whitelist and explains every fallback.
  • Preserve append-only and PK changelog semantics, offsets, checkpoint recovery, projection ordering, and schema evolution.
  • Prove allocator/version compatibility at the Java-to-native Arrow boundary.
  • Add integration and failover tests against the stock Flink Fluss connector as the semantic oracle.
  • Benchmark the direct Arrow path against the existing row-materializing path.
  • Update docs/connectors/fluss.md with the exact accelerated and fallback circumstances.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by checking Apache Fluss issues #4005 and #4006 and confirming that a released artifact provides the public Arrow polling API. Then trace the planner matcher, the stock Flink Fluss connector used as the semantic oracle, and docs/connectors/fluss.md. Done means the whitelist and fallback explanations, integration and failover tests, compatibility checks, benchmark, and documentation updates are complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, documentation, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.