apache / apache/helix

Controller missing events due to disconnect

Open
#3,005 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
504
Forks
255
Avg merge
17d 47m
Merged PRs (30d)
2

Description

### Describe the bug
If the controller's ZkHelixManager is disconnected for longer than the retry timeout (_waitForConnectedTimeout , defaults to 10seconds) but shorter than the negotiated ZK session timeout, then the controller will not react to any events that occurred during that disconnected period.

The controller will only pick up this change after another event triggers a data refresh and rebalance.

### To Reproduce
1. Disconnect ZKHelixManager
2. Kill participant node
3. Reconnect ZKHelixManager after _waitForConnectedTimeout (10s) exceeded but before session timeout (60s) reached
4. Observe that dead participant still has assignments even though offline from ZK view (no LiveInstance node)
5. Trigger another event in cluster and see that dead participant now has replicas moved off.

### Expected behavior
Controller should respond to current state of cluster after reconnect

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.