[client] Expose Arrow changelog batches for primary-key tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Motivation
Fluss already stores ARROW primary-key log batches as Arrow IPC rows plus a per-row change-type vector. The Java client currently exposes row-oriented polling publicly, while record-batch polling is internal and rejects primary-key tables.
StreamFusion needs a Flink source path that can retain Arrow buffers from the Fluss Java client instead of first materializing every live-log record as RowData. Other vectorized consumers have the same need.
### Proposed scope
- Expose record-batch polling on the public `LogScanner` API for ARROW log tables.
- Preserve the stored per-row `ChangeType` values for primary-key changelogs, including `UPDATE_BEFORE`, `UPDATE_AFTER`, and `DELETE`.
- Keep the change-type vector aligned when batches are sliced or truncated at subscribed offsets.
- Preserve explicit Arrow ownership and close semantics.
- Keep existing scanner implementations compatible by providing a default unsupported implementation.
Snapshot reads can continue through the normal row conversion path initially; this issue targets the live log path where Fluss already has the desired Arrow representation.
A caller-provided Arrow allocator would also be useful for consumers such as StreamFusion, but allocator ownership is a broader API/lifecycle decision and can be handled in a follow-up.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the public LogScanner API and the existing internal record-batch polling path. Trace ARROW live-log handling for primary-key tables, especially batch slicing or truncation at subscribed offsets and ownership cleanup. Done means public polling preserves aligned ChangeType values and explicit close semantics while existing scanner implementations remain compatible through a default unsupported implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100