apache / apache/iceberg

Arrow: Add FileAppender<VectorSchemaRoot> for vectorized Parquet writes

Open
#17,748 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Problem

Iceberg's write path (`FileAppender`, `TaskWriter`) accepts only row objects. Projects with Arrow-columnar data (Fluss, Flink, PyIceberg) must materialize `GenericRecord` instances to write Iceberg tables, even when the source data is already in columnar form.

The vectorized read path exists (`iceberg-arrow` module) but there is no write-side equivalent.

### Proposal

Add a `FileAppender` implementation (likely in `iceberg-arrow` or a new `iceberg-arrow-writer` module) that:

1. Accepts Arrow `VectorSchemaRoot` batches
2. Writes to Parquet using Iceberg's existing `ParquetWriter` infrastructure (or `parquet-java`'s Arrow writer once [parquet-java#3733](https://github.com/apache/parquet-java/issues/3733) lands)
3. Collects Iceberg `Metrics` (column sizes, null counts, min/max bounds) from the Arrow vectors
4. Integrates with existing `OutputFileFactory`, `TaskWriter`, and file rolling logic

### Motivation

- **Apache Fluss** — Arrow-native streaming storage that tiers to Iceberg. Currently materializes `GenericRecord` per row ([fluss#4047](https://github.com/apache/fluss/issues/4047)).
- **Apache Flink** — Flink's Iceberg connector serializes internal rows to `Record` objects at the write boundary.
- **PyIceberg / iceberg-rust** — Arrow-native clients (Polars, DataFusion, DuckDB) writing to Iceberg.

### Dependencies

- Can start today using Iceberg's internal `ColumnWriteStore` access (Iceberg's `ParquetWriter` already manages this directly, not through `parquet-java`'s public `ParquetWriter.Builder`)
- Full zero-copy encoding depends on [parquet-java#3733](https://github.com/apache/parquet-java/issues/3733), but an initial implementation iterating Arrow vectors per-value is still a significant improvement over row-object construction

### Scope

- New module or extension in `iceberg-arrow`
- Existing `FileAppender` and `TaskWriter` APIs unchanged
- Arrow dependency remains optional

Contributor guide

Open the contributing guide

Research direction

Start with the existing vectorized read path in the iceberg-arrow module and inspect FileAppender, TaskWriter, ParquetWriter, OutputFileFactory, and TaskWriter integration points. Define the module boundary and compare the Arrow batch flow with existing row-based writing, including ColumnWriteStore access and the parquet-java#3733 dependency. Done means VectorSchemaRoot batches write Parquet, produce Iceberg metrics, and participate in rolling without changing existing Record APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.