Support parquet write from Arrow record batch
- 主要语言
- Java
- 星标
- 94
- 派生
- 152
- 平均合并
- 3 天 16 小时
- 30 天内合并 PR
- 11
描述
### Describe the enhancement requested
I'm working on https://github.com/alibaba/fluss/issues/107 which enable convert Fluss arow structure data to Parquet directly but found the API missing in here.
Althogh [ARROW-11776](https://github.com/apache/arrow-java/commit/0318b07908f240a57f134e34b871786f6a168b6c) supports to write from ArrowReader to file, it read from the ArrowReader, write and close the file direclty. But it's in a very coarse-grained , we almost have no control about the writing. Sometime, we want to control when to close the written parquet. Also it requires `ArrowReader`, but if the arrow RecordBatch is read continuously from remote server . It's not easy to constuct a `ArrowReader`.
So, I think we may need to support the interface to write Arrow RecordBatch to Parquet via [virtual ::arrow::Status WriteRecordBatch(const ::arrow::RecordBatch& batch) = 0](https://github.com/apache/arrow/blob/c506b0806bd2b90410400d349a16bc4a5b1dd51c/cpp/src/parquet/arrow/writer.h#L125)
Just to as a show case, the api may look like:
```
public class ArrowBatchParquetWriter {
void write(RecordBatch recordbatch);
void close()
}
```
贡献指南
调研方向
首先检查现有的基于 ArrowReader 的 Parquet 写入路径以及提议的 ArrowBatchParquetWriter API。将其需求与 Arrow 的 WriteRecordBatch 接口和 RecordBatch 的用法进行比较。完成的标准是,调用方可以增量写入多个记录批次,并且无需 ArrowReader 即可显式关闭 Parquet 输出。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100