apache / apache/parquet-java

Add batch/columnar write API for Arrow VectorSchemaRoot (Java parity with C++/Python)

オープン
#3,733 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
3.1k
フォーク
1.6k
平均マージ
3日 12時間
マージ済み PR(30日)
33

説明

### Problem

`parquet-java` has no public API to write Arrow `VectorSchemaRoot` to Parquet files. Callers must construct row objects and feed them through `ParquetWriter.write(T)` one at a time. Arrow C++ and PyArrow support this natively via `parquet::WriteTable()` / `pq.write_table()`.

Related prior discussion:
- #2264 (open since 2019)
- #3353

### Motivation

Several downstream Java projects work with Arrow-columnar data internally and must materialize row objects solely to satisfy `ParquetWriter`'s input API:

- Apache Iceberg (`FileAppender`) — [iceberg#17748](https://github.com/apache/iceberg/issues/17748)
- Apache Fluss (Arrow-native streaming storage) — [fluss#4047](https://github.com/apache/fluss/issues/4047)
- Apache Paimon (worked around this by building their own `paimon-arrow` writer that bypasses `parquet-java`'s row API entirely)

### Key Insight

For PLAIN-encoded, fixed-width columns, Arrow's in-memory format (contiguous little-endian values) is identical to Parquet's PLAIN page encoding. A zero-copy path is possible by wrapping the Arrow data buffer directly as a `BytesInput` and passing it to `PageWriter.writePage()`.

For nullable columns, Arrow's validity bitmap can be scanned for contiguous non-null runs, with each run bulk-copied and definition levels encoded as RLE runs of the same value — O(null_transitions) instead of O(N).

### Proposed Design

A new `ArrowParquetWriter` in the `parquet-arrow` module that writes pages directly to `PageWriter` rather than going through `RecordConsumer`/`ColumnWriter`:

```
ArrowParquetWriter.writeBatch(VectorSchemaRoot)
→ per column: selects optimal write strategy
→ writes pages to PageWriter via getPageWriter(ColumnDescriptor)
→ manages row groups via ParquetFileWriter
```

Per-column strategy selection (best to worst):

1. **Zero-copy** (non-null + fixed-width + PLAIN): wrap Arrow buffer as `BytesInput` directly. RL/DL as single-value RLE runs. Stats from sequential buffer scan.
2. **Bulk-copy with nulls** (nullable + fixed-width + PLAIN): scan validity bitmap for non-null runs, bulk-copy each, emit DL as RLE runs per null-transition.
3. **Variable-width rewrite** (PLAIN + string/binary): single-pass transformation of Arrow offset+data buffers to Parquet's length-prefixed format.
4. **Dictionary**: map Arrow's `DictionaryEncodedVector` to Parquet dictionary pages, or build dictionary from plain vector.
5. **Fallback**: per-value through `ValuesWriter` (same cost as today, for unsupported encoding/type combos).

All verified public APIs exist for this:
- `PageWriteStore.getPageWriter(ColumnDescriptor)` — get column page writer directly
- `PageWriter.writePage(BytesInput, valueCount, rowCount, stats, encodings)` — write pre-encoded pages
- `BytesInput.from(ByteBuffer)` — zero-copy buffer wrapping
- `RunLengthBitPackingHybridEncoder` — encode RL/DL levels
- `ColumnChunkPageWriteStore.flushToFileWriter()` — flush pages to file

### Why Not Extend ParquetWriter?

`ParquetWriter.write(T)` calls `InternalParquetRecordWriter.write()` which increments `recordCount` by 1 per call and checks row-group boundaries based on that count. This is incompatible with batch semantics. The writer must manage `ParquetFileWriter` and row groups directly.

### Scope

- Parquet file format unchanged
- Existing `ParquetWriter` API unchanged
- `parquet-arrow` gains `parquet-hadoop` as a compile dependency (for `ParquetFileWriter`, `ColumnChunkPageWriteStore`)
- No Hadoop runtime dependency for uncompressed output (compression requires caller-supplied `CompressionCodecFactory`)
- Flat schemas (primitive columns) first; nested types deferred

### Implementation Phases

1. Core infrastructure + zero-copy for non-null fixed-width PLAIN columns
2. Nullable column support (validity bitmap scanning + run-based bulk copy)
3. Variable-width types (string/binary offset rewriting)
4. Dictionary encoding support
5. Bloom filter, nested types, advanced features

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

まず `parquet-arrow` モジュールと、指定された `PageWriteStore`、`PageWriter`、`BytesInput`、`ColumnChunkPageWriteStore` API を読みます。提案されている作業は 5 つの実装フェーズにまたがり、null 非許容の固定幅 PLAIN 列に対するゼロコピー書き込みから始まります。完了の判断は選択したフェーズに基づいて行うものとし、全体の範囲には nullable、可変幅、および dictionary のサポートが含まれます。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
data-engineering
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
30/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。