Druid Coordinator stop assign segments after the zk restart
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Druid 25.0
Zookeeper version: 3.5.10
### Description
1. Zookeeper restart due to the node crash in 05:50 UTC
2. Coordinator log
`2025-03-17T05:50:23,080 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the leader...
2025-03-17T05:50:24,370 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT30S].`
3. There are no new segment assignments, but the following log is present with no error logs:
`Polled and found 201 rule(s) for 193 datasource(s).`
I have attempted the following approaches to recover from the issue:
1. Restarted the coordinator leader and follower, but this did not help.
2. Restarted the Zookeeper follower, but this did not help.
3. Restarted the Zookeeper leader, which resolved the issue.
Contributor guide
Research direction
Start with the DruidCoordinator leader-election and coordination path described in the logs, then trace what happens after ZooKeeper loses and regains leadership. Compare the coordinator, ZooKeeper follower, and ZooKeeper leader restart behavior. Done means segment assignments resume after a ZooKeeper restart without requiring a ZooKeeper leader restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100