[Bug] Flink Dedicated Streaming Compact Job Is Stuck When Its Checkpointed Snapshot Has Expired
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon 1.3.1
### Compute Engine
Flink 1.16
### Minimal reproduce step
1. Start a checkpoint-enabled Flink Dedicated Streaming Compact job.
2. Wait until a checkpoint records snapshot N as the next source cursor.
3. Commit newer snapshots.
4. Explicitly trigger snapshot expiration, or use a non-write-only table whose commit path performs expiration, so that snapshot N is removed.
5. Trigger Flink failover and restore from the checkpoint.
6. Observe repeated OutOfRangeException for snapshot N.
### What doesn't meet your expectations?
A Flink Dedicated Streaming Compact job can become permanently stuck after recovering from a checkpoint whose next snapshot has already been removed by
Paimon snapshot retention.
After recovery, the job continues reading the expired snapshot ID and fails with OutOfRangeException. Flink retries the job from the same checkpoint, so
every restart uses the same invalid snapshot ID. The job never advances to a valid snapshot and cannot self-recover.
Typical error:
OutOfRangeException: Snapshot does not exist or has expired
### Anything else?
No
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Flink Dedicated Streaming Compact job's checkpoint restore path, focusing on how it handles an expired snapshot and the reported OutOfRangeException. Reproduce the failover after snapshot expiration, then verify that recovery no longer retries the same invalid snapshot and can advance to a valid snapshot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100