apache / apache/iceberg

Snapshot Retention Optimization via Spark Action API Deletes Metadata but Retains Data Files — Clarification Needed

Open
#15,611 2 comments 0 reactions 0 assignees View on GitHub
question stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Query engine

Spark Engine

### Question

Hi all,I am utilizing the Spark Action API to implement snapshot retention optimization through the relevant APIs. However, I have observed that while the snapshot metadata files (snap- files) are being deleted as expected, the corresponding data files associated with these snapshots remain intact and are not removed. Could you help me understand the underlying reason for this behavior? Any help would be appreciated.

// Expire snapshots older than current time, retain at least 1
val orphanFileThreshold = java.time.LocalDateTime.now().minusMinutes(2)
.toInstant(java.time.ZoneOffset.UTC)
.toEpochMilli

val expireResult = SparkActions.get()
.expireSnapshots(table).expireOlderThan(orphanFileThreshold)
.retainLast(5).expireFiles()

Regards
Anush

Contributor guide

Open the contributing guide

Research direction

Start with the SparkActions.get().expireSnapshots(table).expireOlderThan(...).retainLast(5).expireFiles() call shown in the report. Inspect how snapshot expiration treats metadata files versus associated data files, then document the reason data files remain and the conditions under which cleanup is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.