apache / apache/paimon

[Bug] Snapshot expiration won't work, space usage and file number continue to grow

Open
#4,743 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.9

### Compute Engine

flink 1.16

### Minimal reproduce step

1. starting a flink job writing to paimon with `snapshot.time-retained` configured.
2. delete some snapshot (>10) between earliest and latest manually
3. after a while, flink job while throw exception frequently like as
![image](https://github.com/user-attachments/assets/f7632d1e-90b3-4f1c-9dac-fe0c6165fa4a)
5. Snapshot expiration won't work anymore, Space usage and the number of files will continue to grow

### What doesn't meet your expectations?

Even if the snapshot sequence is incomplete, Paimon should have a certain level of robustness to complete the snapshot expiration process rather than simply stop functioning.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with a Flink 1.16 job writing to Paimon with snapshot.time-retained configured, then manually delete more than 10 snapshots between the earliest and latest. Trace the snapshot expiration path after the resulting exceptions; done means expiration continues despite incomplete snapshot sequences and space usage and file counts stop growing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.