Possible OutOfRangeException when reading the stream log table after resuming from a long interruption
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
When Flink recovers from a checkpoint after a long interruption, subtasks may throw an OutOfRangeException, as shown in the logs below.
I understand this is caused by local tablet logs being offloaded to remote storage, which makes the recovered split offsets stale and leads to an OutOfRange condition.
While this behavior is acceptable for Kafka-style logs, it is not appropriate for data lake–backed log tables. In this case, the job will keep failing indefinitely. We should handle this scenario explicitly to avoid permanent recovery failures.
### Solution
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start from Flink checkpoint recovery and the OutOfRangeException shown in the issue, then trace how recovered split offsets interact with local tablet logs after offloading to remote storage. Compare the behavior for data lake–backed log tables with Kafka-style logs. Done means a job recovering after a long interruption does not fail indefinitely because its recovered offsets are stale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100