apache / apache/auron

[Feature] Add Native Scan Support for Apache Iceberg MOR Tables

Open
#2,505 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

**Is your feature request related to a problem? Please describe.**

Currently, Auron can accelerate Iceberg scans only when the planned FileScanTasks do not contain delete files. For Iceberg Merge-On-Read tables, Auron falls back to Spark's Iceberg reader instead of using the native execution engine.

This prevents Auron from accelerating common Iceberg V2 MOR workloads, especially tables with row-level deletes, updates.

**Describe the solution you'd like**

Add native Iceberg MOR read support to Auron.

The preferred approach is to keep Iceberg Spark responsible for catalog access, snapshot planning, partition pruning, and producing FileScanTasks, then serialize the planned Iceberg scan tasks to the native engine. The native side should use iceberg-rust to read Iceberg Parquet data files and apply supported delete files correctly.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Spark Iceberg produces and serializes FileScanTasks for the native engine, then inspect the native Iceberg read path and its use of iceberg-rust. Done means Auron can natively scan Iceberg Merge-On-Read tables, including supported delete files, instead of falling back to Spark's reader.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.