apache / apache/druid

Broker retains stale Historical IP after pod recreation; service discovery watch not recovering from stream reset

Open
#19,055 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

## Description

We are encountering an issue where the [Apache Druid](chatgpt://generic-entity?number=0) Broker fails to refresh its service discovery state when Historical pods are recreated with new IP addresses in Kubernetes.

This was initially observed during a cluster upgrade, where there may have been transient connectivity issues with the Kubernetes API server. However, the problem persisted even after the cluster stabilized.

The Broker continues attempting to connect to a stale Historical IP address (which may already be reassigned to a non-Druid pod). The issue does not self-heal and only resolves after manually restarting the Broker pod.

## Environment

- Deployment: Kubernetes
- Discovery: K8s-based node discovery
- Module: extensions-core/kubernetes-extensions
- Components affected: Broker

## Symptoms

- Historical pod is terminated.
- A new Historical pod is created with a new IP.
- Broker continues attempting to connect to the old IP.
- Issue persists indefinitely until Broker pod is restarted manually.

Broker logs repeatedly show:

```
org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Sync failed for server[https://xx:8091/] while [Handling response with code[0], description[null]]. Failed [1] times in the last [0] seconds. (org.jboss.netty.channel.ChannelException: Faulty channel in resource pool)
```

Additionally, errors from the K8s discovery layer:
```
ERROR [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcherpeon]
org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher -
Error while watching role[PEON]
java.lang.RuntimeException: IO Exception during hasNext method.
at io.kubernetes.client.util.Watch.hasNext(Watch.java:183)
at org.apache.druid.k8s.discovery.DefaultK8sApiClient$2.hasNext(DefaultK8sApiClient.java:132)
at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:266)
...
Caused by: okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL
at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:355)
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:281)
at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:200)
at io.kubernetes.client.util.Watch.hasNext(Watch.java:181)
```

## Root Cause

In extensions-core/kubernetes-extensions, the K8sDruidNodeDiscoveryProvider$NodeRoleWatcher encounters:

```
okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL
```

## Expected Behavior

- On watch stream reset or IO exception, the watcher should automatically re-establish the watch.
- A fresh watch should be established
- Node inventory should be reconciled.
- Terminated Historical nodes should be removed from Broker view.

Contributor guide

Open the contributing guide

Research direction

Start with K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching and DefaultK8sApiClient, following how Watch.hasNext failures are handled after an HTTP/2 stream reset. Reproduce or trace the StreamResetException path and verify that a new watch is established, node inventory is reconciled, and terminated Historical nodes disappear from the Broker view.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.