Indexer SelfDiscovery API always returns 503
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Indexer `status/selfDiscoverd` returns 503 even if Indexer is up and running
### Affected Version
0.22.1
### Description
After bringing up the Indexer pods, even if they show up in Druid UI and start executing tasks, the `status/selfDiscovered` API yet returns 503. We use this as the readiness probe in our k8s deployment, so the indexer pods are never considered ready by our k8s because of this issue.
I have port-forwarded localhost:8091 to Indexer pod's port 8091 as follows.
```
hrajendran@harini-rajendran docker % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3997566dbfcc apache/druid:0.21.0 "/druid.sh indexer" 2 minutes ago Up 2 minutes 0.0.0.0:8091->8091/tcp, 0.0.0.0:8106-8110->8106-8110/tcp indexer
```
**Version 0.21.0 [Working as expected]**
```
hrajendran@harini-rajendran docker % curl http://localhost:8091/status/selfDiscovered/status
{"selfDiscovered":true}%
hrajendran@harini-rajendran docker % curl -v http://localhost:8091/status/selfDiscovered
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8091 (#0)
> GET /status/selfDiscovered HTTP/1.1
> Host: localhost:8091
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 17 Mar 2022 17:27:08 GMT
< Content-Type: application/json
< Vary: Accept-Encoding, User-Agent
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0
```
**Version 0.22.1 [Broken]**
```
hrajendran@harini-rajendran docker % curl http://localhost:8091/status/selfDiscovered/status
{"selfDiscovered":false}%
hrajendran@harini-rajendran docker % curl -v http://localhost:8091/status/selfDiscovered
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8091 (#0)
> GET /status/selfDiscovered HTTP/1.1
> Host: localhost:8091
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Date: Thu, 17 Mar 2022 17:24:48 GMT
< Content-Type: application/json
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0
```
Contributor guide
Research direction
Start by reproducing GET /status/selfDiscovered and /status/selfDiscovered/status on Indexer versions 0.21.0 and 0.22.1, using the readiness-probe scenario described. Trace the self-discovery status handling and compare the behavior between versions. Done means a healthy, running Indexer reports selfDiscovered=true and the endpoint returns HTTP 200 rather than 503.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100