apache / apache/iceberg-rust

RFC: File Format API for Apache Iceberg Rust

Open
#2,382 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

This RFC proposes a File Format API for the iceberg Rust crate that decouples Iceberg's read and write paths from any single file format. Today, iceberg-rust can only read and write Parquet data files: the format is hard-wired into ArrowReader, ParquetWriter, and every layer that touches them. The Java project shipped an analogous abstraction (FormatModel) in February 2026 via PR [#12774](https://github.com/apache/iceberg/pull/12774), and PyIceberg has an open proposal ([apache/iceberg-python#3100](https://github.com/apache/iceberg-python/issues/3100)) for the same concept.

### Describe the solution you'd like

The RFC doc below describes how Rust should implement an equivalent capability, specifically a FormatModel trait, a registry, and format-agnostic scan and write paths, using idiomatic Rust constructs: traits with trait objects at the registry boundary, feature flags for compile-time format composition, and RecordBatch as the canonical data type. Because iceberg-rust is pre-1.0, the design takes cleaner tradeoffs than the Java community could: it avoids the wrapper pattern and generic-parameter compromises that were forced on Java by backward-compatibility constraints. The scope is the abstraction layer and its Parquet implementation; ORC, Avro data-file, Vortex, and Lance support are explicitly follow-up work that validates the API's extensibility.

RFC doc PR: https://github.com/apache/iceberg-rust/pull/2384

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start with the RFC document in PR #2384, then inspect the existing ArrowReader and ParquetWriter paths in the iceberg Rust crate. Trace where Parquet is hard-wired and compare the proposed FormatModel, registry, feature flags, and RecordBatch boundaries. Done means the RFC is resolved into an agreed abstraction layer and Parquet implementation; other formats are explicitly follow-up work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.