apache / apache/pinot

Server returns `/health = OK` before server is ready to serve queries

Open
#16,565 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

We've been hit by a bug related to #14529 and external orchestration of Pinot restarts.

Let's assume you have a table with 2 replica groups

- External system sends SIGTERM to pinot-server-1
- pinot-server-1 sets `IS_SHUTDOWN_IN_PROGRESS
- Pinot broker stops routing to pinot-server-1
- pinot-server-1 starts back up with `/health` not returning `OK`
- pinot-server-1 [startupServiceStatusCheck](https://github.com/apache/pinot/blob/642bf00501ef0cc0ddb79ade00b2eff695590ea0/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java#L150) completes.
- *start of problem*: External system seems `/health` return OK
- *problem*: External system restarts pinot-server-2
- *problem*: Queries fail because `pinot-server-1` and `pinot-server-2` are both not serving queries
- `pinot-server-1` sets `IS_SHUTDOWN_IN_PROGRESS` false
- Broker adds `pinot-server-1` back to the routing table, and queries succeed again

In our case, this caused ~17 seconds of down time.

It's not clear how to orchestrate this correctly in Pinot. It seems you have to check the broker routing table for every table to ensure your server is found in there. But there's no clear API for "Is X server available for all necessary segments" or "is Y server going to cause downtime if I take it down". So if you're performing a rolling restart, you're kind of crossing your fingers you wait long enough between replica group restarts.

Contributor guide

Open the contributing guide

Research direction

Start with pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java and the startupServiceStatusCheck flow. Reproduce the restart sequence described in the issue, then inspect how /health relates to broker routing and query availability. Done means the health signal or an accompanying API reliably indicates whether restarting a server can cause downtime.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.