apache / apache/druid

Failing Broker Queries - Faulty Channel

Open
#7,376 5 comments 2 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Broker queries fail with Faulty Channel Exception

### Description

This error occurs when the broker queries fail on MM due to improper shutdown of Peon Task. I have patched this locally by explicitly un-announcing the peon task before destroy is called

This change was applied here https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java#L605

```
curatorDruidNodeAnnouncer.unannounce(new DiscoveryDruidNode(
new DruidNode(
node.getServiceName(),
node.getHost(),
taskInfo.processHolder.port,
taskInfo.processHolder.port,
taskInfo.processHolder.tlsPort,
node.isEnablePlaintextPort(),
node.isEnableTlsPort()
),
DruidNodeDiscoveryProvider.NODE_TYPE_PEON,
ImmutableMap.of()
));

try {
log.info("Destroying task: %s . Sleeping for 10000 millis", taskid);
Thread.sleep(10000);
} catch (InterruptedException e) {
log.error("Sleep interuppedted for task: %s", taskid);
}

taskInfo.processHolder.process.destroy();
```

This is similar to https://github.com/apache/incubator-druid/issues/6043 but the exceptions reported there are slightly different.

Contributor guide

Open the contributing guide

Research direction

Start in indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java around the task-destruction code at the referenced line. Review how Peon task shutdown and node announcement interact, then investigate the faulty-channel broker query failures described here and compare them with issue #6043. Done means broker queries no longer fail because a Peon remains improperly announced during shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.