apache / apache/druid

Unreachable worker nodes may prevent overlord from gaining leadership

Open
#14,387 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Area - Streaming Ingestion Bug Error handling
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

It is possible for middle-manager (and potentially indexers) to be discoverable but somehow unreachable by the overlord.
In that case, when an overlord restarts it will fail to regain leadership while waiting on all the workers to sync.

This problem arises when running with `druid.indexer.runner.type=httpRemote` where the overlord startup sequence relies first syncing state with all the worker nodes. In the event that a worker node is unresponsive or there are network connectivity issues it might make sense for the overlord to instead ignore nodes and proceed with startup, rather than failing to become leader.

### Affected Version

24.0.2

### Description

Attached is a sample output log illustrating how startup might fail.

```
listener becomeLeader() failed. Unable to become leader: {class=org.apache.druid.curator.discovery.CuratorDruidLeaderSelector, exceptionType=class java.lang.RuntimeException, exceptionMessage=java.lang.reflect.InvocationTargetException}

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.druid.indexing.overlord.TaskMaster$1.becomeLeader(TaskMaster.java:161)
at org.apache.druid.curator.discovery.CuratorDruidLeaderSelector$1.isLeader(CuratorDruidLeaderSelector.java:98)
at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:702)
at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:698)
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446)
at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341)
at org.apache.druid.indexing.overlord.TaskMaster$1.becomeLeader(TaskMaster.java:158)
... 7 more
Caused by: java.lang.RuntimeException: org.apache.druid.java.util.common.RE: Failed to sync with worker[middlemanager-0:8091].
at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.start(HttpRemoteTaskRunner.java:285)
... 14 more
Caused by: org.apache.druid.java.util.common.RE: Failed to sync with worker[middlemanager-0:8091].
at org.apache.druid.indexing.overlord.hrtr.WorkerHolder.waitForInitialization(WorkerHolder.java:344)
at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.startWorkersHandling(HttpRemoteTaskRunner.java:560)
at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.start(HttpRemoteTaskRunner.java:265)
... 14 more
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the leadership startup path through TaskMaster, HttpRemoteTaskRunner.startWorkersHandling, and WorkerHolder.waitForInitialization, using the reported stack trace as the entry point. Reproduce the startup failure with an unreachable worker if possible, then verify that the overlord can regain leadership without waiting indefinitely for that worker.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.