apache / apache/paimon

[Bug] Flink Dedicated Streaming Compact Job Is Stuck When Its Checkpointed Snapshot Has Expired

Open
#9,533 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 1.3.1

### Compute Engine

Flink 1.16

### Minimal reproduce step

1. Start a checkpoint-enabled Flink Dedicated Streaming Compact job.
2. Wait until a checkpoint records snapshot N as the next source cursor.
3. Commit newer snapshots.
4. Explicitly trigger snapshot expiration, or use a non-write-only table whose commit path performs expiration, so that snapshot N is removed.
5. Trigger Flink failover and restore from the checkpoint.
6. Observe repeated OutOfRangeException for snapshot N.

### What doesn't meet your expectations?

A Flink Dedicated Streaming Compact job can become permanently stuck after recovering from a checkpoint whose next snapshot has already been removed by
Paimon snapshot retention.

After recovery, the job continues reading the expired snapshot ID and fails with OutOfRangeException. Flink retries the job from the same checkpoint, so
every restart uses the same invalid snapshot ID. The job never advances to a valid snapshot and cannot self-recover.

Typical error:

OutOfRangeException: Snapshot does not exist or has expired

Image

### Anything else?

No

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Flink Dedicated Streaming Compact job's checkpoint restore path, focusing on how it handles an expired snapshot and the reported OutOfRangeException. Reproduce the failover after snapshot expiration, then verify that recovery no longer retries the same invalid snapshot and can advance to a valid snapshot.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.