apache / apache/pulsar

[Bug] ZKSessionWatcher will trigger a ConnectionLost session event if query takes more than 1/15 of zk session time, causing flushing of all caches

Open
#24,674 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

@gaozhangmin During a test related PR #24171 I noticed this about the same code location:

https://github.com/apache/pulsar/blob/6e8c3496647df275071190aec9d29ecbcee55e80/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java#L108-L115

> This will trigger a ConnectionLost session event. That's why a delay should be kept under 2000 ms.
> Currently MockZooKeeper's session timeout is hard coded to 30000ms and the check interval is 1/15 of this, therefore 2000ms.
>
> When there's a test that blocks the ZooKeeper thread for more than 2000ms, it will result in a ConnectionLost event.
> This PR makes the session timeout configurable and it's possible to increase the session timeout for tests where it's necessary to use a longer session timeout to avoid ConnectionLost session events.

This applies likewise to production code. This is problematic since the effective timeout becomes 1/15 of the zk session timeout for Pulsar operations.

Together with this change made in #23018 it will add significant load to ZooKeepers:
https://github.com/apache/pulsar/blob/829df71901de726a11bb3c8ee934d14b67ee9694/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java#L513-L519

I'll file a new issue about resolving this problem.

_Originally posted by @lhotari in https://github.com/apache/pulsar/issues/24624#issuecomment-3227265882_

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java at lines 108-115, then read the related logic in AbstractMetadataStore.java at lines 513-519 and the context from #23018 and #24171. Confirm how the check interval creates an effective timeout of 1/15 of the ZooKeeper session timeout. Done should prevent long operations from causing spurious ConnectionLost events and avoid adding unnecessary ZooKeeper load.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.