Support direct Arrow RecordBatch to Parquet conversion
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
## Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
## Description
This issue tracks the Arrow-to-Parquet conversion part of splitting parent task #437.
We need a reusable utility or writer path that can convert Arrow `RecordBatch` directly into Parquet without first converting through row-oriented representations. This would improve the efficiency and clarity of Arrow-native data flows, especially for tiering and other lakehouse-related write paths.
Possible scope:
- provide a reusable Arrow `RecordBatch` to Parquet conversion utility;
- define the supported Arrow/Parquet type mapping and failure behavior;
- make the conversion path suitable for direct reuse by tiering writers and other lake components.
This is intended to be one sub-task of #437, while enabling tiering source to read data as Arrow `RecordBatch` is tracked separately.
## Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading parent issue #437 and tracing the existing tiering and lakehouse write paths to find where Arrow RecordBatch and Parquet handling belong. Define the supported Arrow-to-Parquet type mapping and failure behavior, then confirm the reusable direct conversion path works for tiering writers and other lake components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100