[kv snapshot lease] Check whether the snapshot is exists when acquireLease in KvSnapshotLeaseMnanager
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Check whether kv snapshot exists is an VERY HEAVY operation, if there is 100K bucket, retain 10 snapshots, this will involve 1 million ZK IO operations! AND IN THE LOCK!
This will soon become a bottleneck of the system and the Flink job.
I think a better solution is to check the availability (and also pin snapshots) via CompletedSnapshotStoreManager. But I think this will involve a lot of code refactoring. I remembered we have discussed this, we have increased the retained snapshot number to 2 (this is the reason we increased it), so it's rarely to happen there is unavailable snapshots when we leasing it. And we can implement the unavailabe check in the future task.
So could you remove the zk snapshot check here, and create an issue for this, and add a TODO comment about the missing check with the issue id.
See comments: https://github.com/apache/fluss/pull/2179#discussion_r2777753842
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate KvSnapshotLeaseMnanager and its acquireLease method, then review the snapshot-existence check and the linked PR 2179 discussion. Remove the ZK snapshot check, add a TODO that references issue 2600, and create the follow-up issue requested for the missing availability and pinning check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100