Snapshot Retention Optimization via Spark Action API Deletes Metadata but Retains Data Files — Clarification Needed
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Query engine
Spark Engine
### Question
Hi all,I am utilizing the Spark Action API to implement snapshot retention optimization through the relevant APIs. However, I have observed that while the snapshot metadata files (snap- files) are being deleted as expected, the corresponding data files associated with these snapshots remain intact and are not removed. Could you help me understand the underlying reason for this behavior? Any help would be appreciated.
// Expire snapshots older than current time, retain at least 1
val orphanFileThreshold = java.time.LocalDateTime.now().minusMinutes(2)
.toInstant(java.time.ZoneOffset.UTC)
.toEpochMilli
val expireResult = SparkActions.get()
.expireSnapshots(table).expireOlderThan(orphanFileThreshold)
.retainLast(5).expireFiles()
Regards
Anush
Contributor guide
Research direction
Start with the SparkActions.get().expireSnapshots(table).expireOlderThan(...).retainLast(5).expireFiles() call shown in the report. Inspect how snapshot expiration treats metadata files versus associated data files, then document the reason data files remain and the conditions under which cleanup is expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, databases
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100