Arrow: Add FileAppender<VectorSchemaRoot> for vectorized Parquet writes
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Problem
Iceberg's write path (`FileAppender`, `TaskWriter`) accepts only row objects. Projects with Arrow-columnar data (Fluss, Flink, PyIceberg) must materialize `GenericRecord` instances to write Iceberg tables, even when the source data is already in columnar form.
The vectorized read path exists (`iceberg-arrow` module) but there is no write-side equivalent.
### Proposal
Add a `FileAppender` implementation (likely in `iceberg-arrow` or a new `iceberg-arrow-writer` module) that:
1. Accepts Arrow `VectorSchemaRoot` batches
2. Writes to Parquet using Iceberg's existing `ParquetWriter` infrastructure (or `parquet-java`'s Arrow writer once [parquet-java#3733](https://github.com/apache/parquet-java/issues/3733) lands)
3. Collects Iceberg `Metrics` (column sizes, null counts, min/max bounds) from the Arrow vectors
4. Integrates with existing `OutputFileFactory`, `TaskWriter`, and file rolling logic
### Motivation
- **Apache Fluss** — Arrow-native streaming storage that tiers to Iceberg. Currently materializes `GenericRecord` per row ([fluss#4047](https://github.com/apache/fluss/issues/4047)).
- **Apache Flink** — Flink's Iceberg connector serializes internal rows to `Record` objects at the write boundary.
- **PyIceberg / iceberg-rust** — Arrow-native clients (Polars, DataFusion, DuckDB) writing to Iceberg.
### Dependencies
- Can start today using Iceberg's internal `ColumnWriteStore` access (Iceberg's `ParquetWriter` already manages this directly, not through `parquet-java`'s public `ParquetWriter.Builder`)
- Full zero-copy encoding depends on [parquet-java#3733](https://github.com/apache/parquet-java/issues/3733), but an initial implementation iterating Arrow vectors per-value is still a significant improvement over row-object construction
### Scope
- New module or extension in `iceberg-arrow`
- Existing `FileAppender` and `TaskWriter` APIs unchanged
- Arrow dependency remains optional
Contributor guide
Research direction
Start with the existing vectorized read path in the iceberg-arrow module and inspect FileAppender, TaskWriter, ParquetWriter, OutputFileFactory, and TaskWriter integration points. Define the module boundary and compare the Arrow batch flow with existing row-based writing, including ColumnWriteStore access and the parquet-java#3733 dependency. Done means VectorSchemaRoot batches write Parquet, produce Iceberg metrics, and participate in rolling without changing existing Record APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100