Support parquet write from Arrow record batch
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the enhancement requested
I'm working on https://github.com/alibaba/fluss/issues/107 which enable convert Fluss arow structure data to Parquet directly but found the API missing in here.
Althogh [ARROW-11776](https://github.com/apache/arrow-java/commit/0318b07908f240a57f134e34b871786f6a168b6c) supports to write from ArrowReader to file, it read from the ArrowReader, write and close the file direclty. But it's in a very coarse-grained , we almost have no control about the writing. Sometime, we want to control when to close the written parquet. Also it requires `ArrowReader`, but if the arrow RecordBatch is read continuously from remote server . It's not easy to constuct a `ArrowReader`.
So, I think we may need to support the interface to write Arrow RecordBatch to Parquet via [virtual ::arrow::Status WriteRecordBatch(const ::arrow::RecordBatch& batch) = 0](https://github.com/apache/arrow/blob/c506b0806bd2b90410400d349a16bc4a5b1dd51c/cpp/src/parquet/arrow/writer.h#L125)
Just to as a show case, the api may look like:
```
public class ArrowBatchParquetWriter {
void write(RecordBatch recordbatch);
void close()
}
```
Contributor guide
Research direction
Start by reviewing the existing ArrowReader-based Parquet writing path and the proposed ArrowBatchParquetWriter API. Compare its needs with Arrow's WriteRecordBatch interface and RecordBatch usage. Done means callers can write multiple record batches incrementally and explicitly close the Parquet output without requiring an ArrowReader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100