Support parquet write from Arrow record batch
- Linguagem predominante
- Java
- Estrelas
- 94
- Forks
- 152
- Merge médio
- 3d 16h
- PRs com merge (30d)
- 11
Descrição
### Describe the enhancement requested
I'm working on https://github.com/alibaba/fluss/issues/107 which enable convert Fluss arow structure data to Parquet directly but found the API missing in here.
Althogh [ARROW-11776](https://github.com/apache/arrow-java/commit/0318b07908f240a57f134e34b871786f6a168b6c) supports to write from ArrowReader to file, it read from the ArrowReader, write and close the file direclty. But it's in a very coarse-grained , we almost have no control about the writing. Sometime, we want to control when to close the written parquet. Also it requires `ArrowReader`, but if the arrow RecordBatch is read continuously from remote server . It's not easy to constuct a `ArrowReader`.
So, I think we may need to support the interface to write Arrow RecordBatch to Parquet via [virtual ::arrow::Status WriteRecordBatch(const ::arrow::RecordBatch& batch) = 0](https://github.com/apache/arrow/blob/c506b0806bd2b90410400d349a16bc4a5b1dd51c/cpp/src/parquet/arrow/writer.h#L125)
Just to as a show case, the api may look like:
```
public class ArrowBatchParquetWriter {
void write(RecordBatch recordbatch);
void close()
}
```
Guia de contribuição
Direção de pesquisa
Comece revisando o caminho existente de gravação de Parquet baseado em ArrowReader e a API proposta de ArrowBatchParquetWriter. Compare suas necessidades com a interface WriteRecordBatch do Arrow e o uso de RecordBatch. O trabalho estará concluído quando os chamadores puderem gravar vários lotes de registros incrementalmente e fechar explicitamente a saída Parquet sem exigir um ArrowReader.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- java
- Domínio
- data-engineering
- Tipo de issue
- Funcionalidade
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Status de atividade
- Estagnada
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 35/100