[Bug] ZKSessionWatcher will trigger a ConnectionLost session event if query takes more than 1/15 of zk session time, causing flushing of all caches
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
@gaozhangmin During a test related PR #24171 I noticed this about the same code location:
https://github.com/apache/pulsar/blob/6e8c3496647df275071190aec9d29ecbcee55e80/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java#L108-L115
> This will trigger a ConnectionLost session event. That's why a delay should be kept under 2000 ms.
> Currently MockZooKeeper's session timeout is hard coded to 30000ms and the check interval is 1/15 of this, therefore 2000ms.
>
> When there's a test that blocks the ZooKeeper thread for more than 2000ms, it will result in a ConnectionLost event.
> This PR makes the session timeout configurable and it's possible to increase the session timeout for tests where it's necessary to use a longer session timeout to avoid ConnectionLost session events.
This applies likewise to production code. This is problematic since the effective timeout becomes 1/15 of the zk session timeout for Pulsar operations.
Together with this change made in #23018 it will add significant load to ZooKeepers:
https://github.com/apache/pulsar/blob/829df71901de726a11bb3c8ee934d14b67ee9694/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java#L513-L519
I'll file a new issue about resolving this problem.
_Originally posted by @lhotari in https://github.com/apache/pulsar/issues/24624#issuecomment-3227265882_
Contributor guide
Research direction
Start with pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java at lines 108-115, then read the related logic in AbstractMetadataStore.java at lines 513-519 and the context from #23018 and #24171. Confirm how the check interval creates an effective timeout of 1/15 of the ZooKeeper session timeout. Done should prevent long operations from causing spurious ConnectionLost events and avoid adding unnecessary ZooKeeper load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100