[lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Paimon's lake writer implements `SupportsRecordBatchWrite` (landed in #3417/#3418), enabling direct Arrow batch to Parquet writes for append-only tables. This avoids the row-by-row serialization overhead through `GenericRecord` objects.
The Iceberg lake writer (`IcebergLakeWriter`) currently writes row-by-row via `GenericAppenderFactory`, resulting in significantly lower throughput for append-only log table tiering compared to the Paimon path.
Since Fluss uses Arrow as its internal columnar format, and both Iceberg and Parquet have native Arrow support (`iceberg-arrow` module, `parquet-arrow` module), we can bridge the Arrow batch directly to Iceberg Parquet files.
### Solution
1. Make `IcebergLakeWriter` implement `SupportsRecordBatchWrite`
2. Create an Arrow batch writer that takes `ArrowBatchData` and writes to Iceberg Parquet files
3. Only enabled for append-only tables (same restriction as Paimon)
4. Preserve Iceberg file metrics tracking for proper table maintenance
Related: #437 (generic arrow-to-parquet issue)
### Upstream Issues for True Zero-Copy Path
The current PR (#4048) implements `SupportsRecordBatchWrite` by reading Arrow vectors and producing `GenericRecord` objects. This avoids binary log deserialization but still materializes row objects because Iceberg and parquet-java lack columnar write APIs.
Upstream issues filed to enable true zero-copy Arrow-to-Parquet writes:
1. **[apache/parquet-java#3733](https://github.com/apache/parquet-java/issues/3733)** — Add `ParquetWriter` to the `parquet-arrow` module. This is the foundational primitive.
2. **[apache/iceberg#17748](https://github.com/apache/iceberg/issues/17748)** — Add `FileAppender` to Iceberg's arrow module. Depends on (or works alongside) the parquet-java change.
Once these land, Fluss can replace the `IcebergArrowBatchHelper` (GenericRecord bridge) with a direct call to Iceberg's vectorized appender.
### Anything else?
All necessary primitives exist in upstream Iceberg (`iceberg-arrow` 1.10.1) and Apache Parquet (`parquet-arrow`). No upstream changes needed for the initial implementation; upstream changes enable the optimized path.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with IcebergLakeWriter and compare its GenericAppenderFactory path with the Paimon SupportsRecordBatchWrite implementation. Trace ArrowBatchData through IcebergArrowBatchHelper, then verify append-only restrictions and Iceberg file metrics; PR #4048 already describes an implementation to compare against.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100