apache / apache/druid

Druid Coordinator stop assign segments after the zk restart

Open
#17,807 1 comment 0 reactions 0 assignees View on GitHub
Area - ZooKeeper/Curator Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Druid 25.0
Zookeeper version: 3.5.10

### Description
1. Zookeeper restart due to the node crash in 05:50 UTC
2. Coordinator log
`2025-03-17T05:50:23,080 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the leader...
2025-03-17T05:50:24,370 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT30S].`
3. There are no new segment assignments, but the following log is present with no error logs:
`Polled and found 201 rule(s) for 193 datasource(s).`

I have attempted the following approaches to recover from the issue:

1. Restarted the coordinator leader and follower, but this did not help.
2. Restarted the Zookeeper follower, but this did not help.
3. Restarted the Zookeeper leader, which resolved the issue.

Contributor guide

Open the contributing guide

Research direction

Start with the DruidCoordinator leader-election and coordination path described in the logs, then trace what happens after ZooKeeper loses and regains leadership. Compare the coordinator, ZooKeeper follower, and ZooKeeper leader restart behavior. Done means segment assignments resume after a ZooKeeper restart without requiring a ZooKeeper leader restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.