apache / apache/auron

Support Iceberg

Open
#1,472 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

### **Is your feature request related to a problem? Please describe.**

Currently, Auron does not support the Iceberg table format, which causes interoperability issues when trying to read or write data stored in Iceberg tables.
In many data lake environments, Iceberg has become a standard format due to its support for schema evolution, partition pruning, and ACID guarantees.
Without Iceberg support, Auron users have to manually convert data or rely on external Spark/Flink jobs, which increases operational complexity.

### **Describe the solution you'd like**

**Resolved scenarios**
The current issue focuses on supporting Spark scenarios and integration with Iceberg. In the future, once Flink support is improved, Flink integration scenarios can also be incorporated.

**Iceberg Support Spark Scenarios**
Implementing Iceberg Spark scenarios, with the core scenarios as follows:

1. Supported Spark Versions:
3.4 / 3.5 / 4.0
3. Supported Write Scenarios:
MERGE INTO, DELETE, UPDATE, APPEND, OVERWRITE (static write), OVERWRITE (dynamic write)
The corresponding physical execution plans are as follows:
a. AppendDataExec: append write
b. OverwriteByExpressionExec: overwrite static partition write
c. OverwritePartitionsDynamicExec: overwrite dynamic partition write
d. ReplaceDataExec: write for COW table in MERGE INTO / DELETE / UPDATE scenarios
e. WriteDeltaExec: write for MOR table in MERGE INTO / DELETE / UPDATE scenarios

Notes:
a.AppendDataExec, OverwriteByExpressionExec, and OverwritePartitionsDynamicExec are implemented in the Spark project.
b.ReplaceDataExec and WriteDeltaExec are implemented in the Iceberg project for Spark 3.4, and have been merged into the Spark project starting from Spark 3.5.

3. Supported Read Scenario:
a.BatchScanExec, which supports reading both COW and MOR tables.
b.This execution plan is implemented in the Spark project.

4. Other Iceberg Execution Plans:
a.CallExec: executes user-defined stored procedures
b.MergeRowsExec: auxiliary execution plan used for MERGE INTO, UPDATE, and DELETE operations

**The core functionality addressed by this issue**
Support for `AppendDataExec` / `OverwriteByExpressionExec` / `OverwritePartitionsDynamicExec` / `ReplaceDataExec` / `WriteDeltaExec` / `BatchScanExec`, with these physical execution plans being converted to native execution.

### **Additional context**

After this feature is implemented, Auron will be able to read Iceberg tables by converting Spark/Iceberg physical execution plans into native Auron execution plans.

### **TODO List**

- [1] Add the Auron-Iceberg project, integrating Iceberg version 1.9.2. 【DONE】
- [2] Integrate the Iceberg module with Spark 3.4 and Spark 3.5 【DONE】
- [3] Add `IcebergConvertProvider` to separately convert Iceberg-specific execution plans.
- [4] Support converting `BatchScanExec` to native.
- [5] Support converting `AppendDataExec` to native.
- [6] Support converting `OverwriteByExpressionExec` to native.
- [7] Support converting `OverwritePartitionsDynamicExec` to native.
- [8] Support converting `ReplaceDataExec` to native.
- [9] Support converting `WriteDeltaExec` to native.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.