apache / apache/paimon

[Bug] remove_orphan_files may accidentally delete data in multitasking concurrent scenarios

Open
#7,710 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

version 1.2

### Compute Engine

Spark 3.5.6

### Minimal reproduce step

table max retained snapshot is 1
'snapshot.num-retained.max' = '1'

2026-04-25 04:37:12,116, Task 1 completed sys.compact operation and submitted snapshot 4959
2026-04-25 04:37:13,195, Task 1 completed snapshot expiration. Mark snapshot 4958 as expired and delete it
2026-04-25 04:37:37,605, Task 2 completed writing and submitted snapshot 4960
2026-04-25 04:37:38,185, Task 2 completed snapshot expiration. Mark snapshot 4959 as expired and delete it
At the same time, Task 1 started executing the remove_orphan_files, but the current snapshot 4959 was not read because 4959 had already been deleted by Task 2, resulting in usedFiles being empty, which in turn caused deleted=candidates, meaning that all files written one day ago were marked as files to be deleted and ultimately deleted.

### What doesn't meet your expectations?

Incorrect deletion of historical data

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the remove_orphan_files flow and the concurrent snapshot-expiration sequence described for Paimon 1.2 on Spark 3.5.6. Trace how snapshots 4958–4960 determine usedFiles and deletion candidates during concurrent tasks. Done means historical data remains intact when expiration and orphan-file removal overlap.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.