apache / apache/fluss

Support reading tiering source data as Arrow RecordBatch

Open
#2,962 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

## Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

## Description
This issue tracks the tiering-source part of splitting parent task #437.

Today, tiering source reads Fluss log data and converts it into downstream storage formats through row-oriented or storage-specific paths. To support a cleaner and more efficient Arrow-based pipeline, tiering source should be able to read data directly as Arrow `RecordBatch`.

This work would provide a reusable Arrow-native read path for tiering, and would also serve as the foundation for directly writing tiered data into Parquet in a later step.

Possible scope:
- add a tiering-source path that reads log data as Arrow `RecordBatch`;
- define the batch lifecycle/ownership clearly to avoid Arrow memory leaks;
- make the Arrow batch path reusable by downstream tiering writers.

This is intended to be one sub-task of #437, while the Arrow-to-Parquet conversion itself is tracked separately.

## Willingness to contribute
- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the tiering-source read path described here and parent task #437, comparing its row-oriented and storage-specific conversions. Define the Arrow RecordBatch ownership and lifecycle, then verify that the reusable batch path can feed downstream tiering writers without leaks; Arrow-to-Parquet conversion is out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.