apache / apache/fluss

[kv snapshot lease] Check whether the snapshot is exists when acquireLease in KvSnapshotLeaseMnanager

Open
#2,600 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Check whether kv snapshot exists is an VERY HEAVY operation, if there is 100K bucket, retain 10 snapshots, this will involve 1 million ZK IO operations! AND IN THE LOCK!

This will soon become a bottleneck of the system and the Flink job.

I think a better solution is to check the availability (and also pin snapshots) via CompletedSnapshotStoreManager. But I think this will involve a lot of code refactoring. I remembered we have discussed this, we have increased the retained snapshot number to 2 (this is the reason we increased it), so it's rarely to happen there is unavailable snapshots when we leasing it. And we can implement the unavailabe check in the future task.

So could you remove the zk snapshot check here, and create an issue for this, and add a TODO comment about the missing check with the issue id.

See comments: https://github.com/apache/fluss/pull/2179#discussion_r2777753842

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate KvSnapshotLeaseMnanager and its acquireLease method, then review the snapshot-existence check and the linked PR 2179 discussion. Remove the ZK snapshot check, add a TODO that references issue 2600, and create the follow-up issue requested for the missing availability and pinning check.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.