apache / apache/datafusion-comet

Iceberg Table Maintenance: Acceleration Opportunities

Open
#3,371 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

We should explore and benchmark Iceberg table maintenance operations, while checking for any missing DataFusion implementations that could accelerate Spark Iceberg table maintenance.

### Describe the potential solution

_No response_

### Additional context

Iceberg table maintenance using Spark: https://iceberg.apache.org/docs/latest/maintenance/#delete-orphan-files

Medium blog compaction with datafusion + rust : https://medium.com/@amitgil87/building-a-distributed-compaction-engine-for-apache-iceberg-with-rust-datafusion-09faa63c1438

Some old discussion to refer : https://github.com/apache/iceberg-rust/issues/624

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.