apache / apache/helix

Legacy IZkStateListenerI0ItecImpl causes zombie participants by discarding session IDs

Open
#3,063 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
504
Forks
255
Avg merge
17d 47m
Merged PRs (30d)
2

Description

### Describe the bug
Helix participants can enter a "zombie" state where they appear healthy in ZooKeeper but are functionally disconnected from cluster operations, unable to process state transition messages. This occurs due to a critical flaw in the legacy compatibility wrapper IZkStateListenerI0ItecImpl that completely ignores session ID parameters when processing ZooKeeper session events.
When the single-threaded ZkEventThread becomes blocked (e.g., during long-running FINALIZE operations), multiple SyncConnected events accumulate in the event queue. Once processing resumes, these events are handled in FIFO order, but the legacy wrapper discards the original session ID from each queued event and falls back to using the current active session ID via getSessionId(). This causes session events originally queued for older sessions to be processed using the context of the most recent session, violating the intended event processing timeline.
The result is catastrophic: the first misprocessed event successfully creates a LiveInstance for the current session, but all subsequent events in the backlog attempt to create LiveInstances for the same current session and fail with "already has a live-instance" exceptions. Each failed attempt partially resets the participant's message handlers (setting _ready = false) but never completes the re-initialization process, leaving the participant in a broken state where it cannot process any state transition messages despite appearing active to the Helix controller.

### To Reproduce
Create a DedicateZkClient, pass any fake sessionID in handleNewSession. It will totally discard that sessionID and will use the current active zk connection session ID and will all the processing based on that. Trigger another handleNewSession event with fake sessionID, it will fail to create liveinstance this time, and handlers will be kept in reset state.

### Expected behavior
Session events should be processed with their original session context regardless of timing delays. The handleNewSession() method should receive and use the correct session ID that was active when the event was originally queued, ensuring proper FIFO event processing and maintaining participant message handling capabilities.

### Additional context
Add any other context about the problem here.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the legacy IZkStateListenerI0ItecImpl wrapper and DedicateZkClient.handleNewSession, then trace how queued events from ZkEventThread pass session IDs into LiveInstance handling. Reproduce the issue with two fake session IDs and verify that each event uses its original session context without leaving handlers reset or creating duplicate live instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.