[Feature] Add Native Scan Support for Apache Iceberg MOR Tables
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 22
Description
**Is your feature request related to a problem? Please describe.**
Currently, Auron can accelerate Iceberg scans only when the planned FileScanTasks do not contain delete files. For Iceberg Merge-On-Read tables, Auron falls back to Spark's Iceberg reader instead of using the native execution engine.
This prevents Auron from accelerating common Iceberg V2 MOR workloads, especially tables with row-level deletes, updates.
**Describe the solution you'd like**
Add native Iceberg MOR read support to Auron.
The preferred approach is to keep Iceberg Spark responsible for catalog access, snapshot planning, partition pruning, and producing FileScanTasks, then serialize the planned Iceberg scan tasks to the native engine. The native side should use iceberg-rust to read Iceberg Parquet data files and apply supported delete files correctly.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by tracing how Spark Iceberg produces and serializes FileScanTasks for the native engine, then inspect the native Iceberg read path and its use of iceberg-rust. Done means Auron can natively scan Iceberg Merge-On-Read tables, including supported delete files, instead of falling back to Spark's reader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100