apache / apache/iceberg-rust

Add `RewriteFiles` support to iceberg-rust

Open
#1,607 9 comments 13 reactions 1 assignee Claimed by @CTTY View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Currently, `iceberg-rust` does not support committing operations that both add and delete files. This feature, known as `RewriteFiles` in `iceberg-java`, is crucial for our use case, particularly for compaction.

### Describe the solution you'd like

Based on my initial investigation in #1606, I believe the following three components are necessary to implement `RewriteFiles` in `iceberg-rust`:

- **Process delete files**: We need to add logic to handle delete files when generating snapshots.
- **Implement `RewriteFilesAction`**: This will require creating a new transaction action, `RewriteFilesAction`, and related classes like `RewriteFilesOperation`.
- **Support validation**: We need to add validation for rewrite operations to prevent conflicts. Currently we don't have any validation mechnism in the codebase, so a new trait `SnapshotValidator` would be needed.

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.