apache / apache/datafusion-comet
Iceberg Table Maintenance: Acceleration Opportunities
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
We should explore and benchmark Iceberg table maintenance operations, while checking for any missing DataFusion implementations that could accelerate Spark Iceberg table maintenance.
### Describe the potential solution
_No response_
### Additional context
Iceberg table maintenance using Spark: https://iceberg.apache.org/docs/latest/maintenance/#delete-orphan-files
Medium blog compaction with datafusion + rust : https://medium.com/@amitgil87/building-a-distributed-compaction-engine-for-apache-iceberg-with-rust-datafusion-09faa63c1438
Some old discussion to refer : https://github.com/apache/iceberg-rust/issues/624
Contributor guide
Assessment
This issue has not been assessed yet.