Failing Broker Queries - Faulty Channel
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Broker queries fail with Faulty Channel Exception
### Description
This error occurs when the broker queries fail on MM due to improper shutdown of Peon Task. I have patched this locally by explicitly un-announcing the peon task before destroy is called
This change was applied here https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java#L605
```
curatorDruidNodeAnnouncer.unannounce(new DiscoveryDruidNode(
new DruidNode(
node.getServiceName(),
node.getHost(),
taskInfo.processHolder.port,
taskInfo.processHolder.port,
taskInfo.processHolder.tlsPort,
node.isEnablePlaintextPort(),
node.isEnableTlsPort()
),
DruidNodeDiscoveryProvider.NODE_TYPE_PEON,
ImmutableMap.of()
));
try {
log.info("Destroying task: %s . Sleeping for 10000 millis", taskid);
Thread.sleep(10000);
} catch (InterruptedException e) {
log.error("Sleep interuppedted for task: %s", taskid);
}
taskInfo.processHolder.process.destroy();
```
This is similar to https://github.com/apache/incubator-druid/issues/6043 but the exceptions reported there are slightly different.
Contributor guide
Research direction
Start in indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java around the task-destruction code at the referenced line. Review how Peon task shutdown and node announcement interact, then investigate the faulty-channel broker query failures described here and compare them with issue #6043. Done means broker queries no longer fail because a Peon remains improperly announced during shutdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100