[Feature] Avoid release all of the LookupFile during failover
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
The LookupFile is managed by the IOManager of the Flink. If job failover, all the cache file will be cleaned. It should be avoid. Because the cache build process is heavy.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue concerns LookupFile cleanup by Flink's IOManager during job failover, but it names no source files, tests, or implementation plan. Start by locating LookupFile and the IOManager integration, then trace what happens during failover. Done would mean preserving the cache files across failover without breaking cleanup, with regression coverage for the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100