apache / apache/paimon

[Bug] Concurrent tag creation and snapshot expiration may delete files referenced by the new tag

Open
#9,220 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the issues and found no issue covering this exact create-tag versus snapshot-expiration race.

### Paimon version

- 1.3.1
- The same race is still present on master commit 54e64e364c1ef62206c173b4fec06a15320b0d32.

### Compute Engine

Java API. The problem is in paimon-core and is compute-engine independent.

### Minimal reproduce step

Preconditions:

1. Create several snapshots.
2. Choose an old snapshot S inside the range that will be expired.
3. Ensure a later snapshot has deleted or replaced files that are still visible in S.
4. Run tag creation and snapshot expiration concurrently.

A deterministic test can inject a latch immediately after ExpireSnapshotsImpl collects tagged snapshots:

1. Thread A enters ExpireSnapshotsImpl.expireUntil.
2. Thread A executes tagManager.taggedSnapshots in 1.3.1, or collectTaggedSnapshots on master, and is paused. Snapshot S has no tag at this point.
3. Thread B calls table.createTag with tag name tag-s and snapshot id S. The tag file is written successfully.
4. Thread B returns success.
5. Resume Thread A.
6. Thread A continues data-file and manifest cleanup using the old tagged-snapshot list, which does not contain tag-s.
7. Thread A finally deletes the snapshot metadata for S.
8. Read tag-s using tag time travel.

The tag file remains present, but its manifest list, manifest, or data files may have been deleted, causing FileNotFoundException or an unreadable tag.

The relevant ordering in release 1.3.1 is:

- ExpireSnapshotsImpl reads tagManager.taggedSnapshots once.
- The same fixed list is used by createDataFileSkipperForTags and findSkippingTags.
- cleanUnusedDataFiles and cleanUnusedManifests run.
- Snapshot metadata is deleted last.

Tag creation independently does:

- AbstractFileStoreTable.findSnapshot loads S.
- TagManager.createTag writes the tag metadata through FileIO.overwriteFileUtf8.

There is no shared lock, GC epoch, revalidation, or second tag collection between these paths.

Current master still collects tagged snapshots once before planning and deleting data and manifest files, and deletes snapshot metadata last.

### What does not meet your expectations?

A successfully created tag should remain readable and must protect every file referenced by its tagged snapshot.

Snapshot expiration currently protects only tags visible when expiration initially collects tag metadata. A tag created after that point can return success while the same expiration operation deletes files referenced by it. This is a TOCTOU race that can cause data loss from the perspective of the new tag.

Checking snapshot existence before and after creating the tag is not sufficient because snapshot metadata is deliberately deleted after data and manifest cleanup. Both checks can succeed before the expiration operation removes the snapshot metadata and referenced files.

### Are you willing to submit a PR?

Not committed at this time.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ExpireSnapshotsImpl.expireUntil and trace how taggedSnapshots or collectTaggedSnapshots feeds createDataFileSkipperForTags, findSkippingTags, cleanUnusedDataFiles, and cleanUnusedManifests. Compare this with AbstractFileStoreTable.findSnapshot and TagManager.createTag, then add the deterministic latch scenario described in the issue. Done means a tag created during expiration remains readable and protects all files referenced by its snapshot.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.