apache / apache/iceberg

best-effort file deletion in ExpireSnapshotsSparkAction when a referenced metadata file is missing

Open
#17,715 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

Feature: best-effort file deletion in ExpireSnapshotsSparkAction when a referenced metadata file is missing

Today, ExpireSnapshotsSparkAction aborts the entire operation if a referenced manifest or manifest-list file is missing from storage (NotFoundException). The failure happens while computing the set of files to delete, so no cleanup runs at all — even for the files it could have safely deleted.
To let expiration make progress instead of failing outright, while staying safe:
- If the missing manifest is reachable only from snapshots being expired, delete every file that was fully resolved, then surface a single aggregated error/warning listing the missing files. Under-deleting here only leaves orphans, which remove_orphan_files can clean up later — it never removes live data.
- If the missing manifest is reachable from a retained snapshot, the delete set can't be computed safely (risk of deleting live files), so fail fast before deleting anything.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start at ExpireSnapshotsSparkAction and trace how the delete set is computed when a manifest or manifest-list file is missing. Verify the retained-snapshot case fails before deletion, while expired-only missing files allow resolved files to be deleted and produce one aggregated warning or error. Done means both safety paths are covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.