apache / apache/fluss

Support direct Arrow RecordBatch to Parquet conversion

Open
#2,963 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

## Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

## Description
This issue tracks the Arrow-to-Parquet conversion part of splitting parent task #437.

We need a reusable utility or writer path that can convert Arrow `RecordBatch` directly into Parquet without first converting through row-oriented representations. This would improve the efficiency and clarity of Arrow-native data flows, especially for tiering and other lakehouse-related write paths.

Possible scope:
- provide a reusable Arrow `RecordBatch` to Parquet conversion utility;
- define the supported Arrow/Parquet type mapping and failure behavior;
- make the conversion path suitable for direct reuse by tiering writers and other lake components.

This is intended to be one sub-task of #437, while enabling tiering source to read data as Arrow `RecordBatch` is tracked separately.

## Willingness to contribute
- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading parent issue #437 and tracing the existing tiering and lakehouse write paths to find where Arrow RecordBatch and Parquet handling belong. Define the supported Arrow-to-Parquet type mapping and failure behavior, then confirm the reusable direct conversion path works for tiering writers and other lake components.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.