apache / apache/druid

Indexer SelfDiscovery API always returns 503

Open
#12,343 3 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Indexer `status/selfDiscoverd` returns 503 even if Indexer is up and running

### Affected Version

0.22.1

### Description
After bringing up the Indexer pods, even if they show up in Druid UI and start executing tasks, the `status/selfDiscovered` API yet returns 503. We use this as the readiness probe in our k8s deployment, so the indexer pods are never considered ready by our k8s because of this issue.

I have port-forwarded localhost:8091 to Indexer pod's port 8091 as follows.
```
hrajendran@harini-rajendran docker % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3997566dbfcc apache/druid:0.21.0 "/druid.sh indexer" 2 minutes ago Up 2 minutes 0.0.0.0:8091->8091/tcp, 0.0.0.0:8106-8110->8106-8110/tcp indexer
```
**Version 0.21.0 [Working as expected]**
```
hrajendran@harini-rajendran docker % curl http://localhost:8091/status/selfDiscovered/status
{"selfDiscovered":true}%

hrajendran@harini-rajendran docker % curl -v http://localhost:8091/status/selfDiscovered
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8091 (#0)
> GET /status/selfDiscovered HTTP/1.1
> Host: localhost:8091
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 17 Mar 2022 17:27:08 GMT
< Content-Type: application/json
< Vary: Accept-Encoding, User-Agent
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0
```

**Version 0.22.1 [Broken]**
```
hrajendran@harini-rajendran docker % curl http://localhost:8091/status/selfDiscovered/status
{"selfDiscovered":false}%

hrajendran@harini-rajendran docker % curl -v http://localhost:8091/status/selfDiscovered
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8091 (#0)
> GET /status/selfDiscovered HTTP/1.1
> Host: localhost:8091
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Date: Thu, 17 Mar 2022 17:24:48 GMT
< Content-Type: application/json
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing GET /status/selfDiscovered and /status/selfDiscovered/status on Indexer versions 0.21.0 and 0.22.1, using the readiness-probe scenario described. Trace the self-discovery status handling and compare the behavior between versions. Done means a healthy, running Indexer reports selfDiscovered=true and the endpoint returns HTTP 200 rather than 503.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.