apache / apache/beam

[Feature Request]: Add a new writer to output Row directly to a parquet file

Open
#23,368 5 comments 0 reactions 0 assignees View on GitHub
io java new feature P2 parquet
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

For now, ParquetIO.sink must convert a `org.apache.beam.sdk.values.Row` to a GenericRecord and then use `AvroParquetWriter` to output records to a parquet file.
I think we can add a new writer, called `RowParquetWriter`, to write a Row directly to parquet files without conversions, which will be more efficient and useful in a SqlTransform case

### Issue Priority

Priority: 2

### Issue Component

Component: io-java-parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.