Support reading tiering source data as Arrow RecordBatch
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
## Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
## Description
This issue tracks the tiering-source part of splitting parent task #437.
Today, tiering source reads Fluss log data and converts it into downstream storage formats through row-oriented or storage-specific paths. To support a cleaner and more efficient Arrow-based pipeline, tiering source should be able to read data directly as Arrow `RecordBatch`.
This work would provide a reusable Arrow-native read path for tiering, and would also serve as the foundation for directly writing tiered data into Parquet in a later step.
Possible scope:
- add a tiering-source path that reads log data as Arrow `RecordBatch`;
- define the batch lifecycle/ownership clearly to avoid Arrow memory leaks;
- make the Arrow batch path reusable by downstream tiering writers.
This is intended to be one sub-task of #437, while the Arrow-to-Parquet conversion itself is tracked separately.
## Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the tiering-source read path described here and parent task #437, comparing its row-oriented and storage-specific conversions. Define the Arrow RecordBatch ownership and lifecycle, then verify that the reusable batch path can feed downstream tiering writers without leaks; Arrow-to-Parquet conversion is out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100