Broker routing can briefly reference segments that servers no longer serve
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Several Pinot workflows rely on broker routing converging with server-side segment state changes before the old serving path becomes unavailable. A stale routing entry can send a query to a server after that server has dropped the segment, producing SERVER_SEGMENT_MISSING.
1. Server readiness versus broker readiness
[Issue #16565](https://github.com/apache/pinot/issues/16565) describes a server returning healthy before brokers had resumed routing to it. During a rolling restart, this allowed the next server to be restarted while neither replica was queryable.
2. Segment deletion / retention
Deletion has the inverse ordering requirement. The controller updates IdealState, servers asynchronously process the resulting transition and remove the segment, and brokers asynchronously rebuild routing. If a broker routes a query using the old assignment after the server has removed the segment, the query fails.
This is less likely for ordinary retention when time pruning excludes expired segments, but it remains possible for queries spanning the retention boundary and for explicit segment deletion.
3. Rebalance transition
Even with add-before-remove rebalance, a query can be dispatched immediately before the source server’s state transition reaches broker routing. If the source has removed the segment by request handling time. We have observed the following errors at least 4 times in a month due to this:
```
Query processing exceptions:
{235=1 segments [] missing on server: }
```
Contributor guide
Research direction
Start by tracing broker routing alongside controller IdealState updates and server-side segment state changes. Reproduce the rolling-restart, deletion, or rebalance timing described in the issue and inspect the SERVER_SEGMENT_MISSING failure. Done means routing no longer dispatches queries to servers after they stop serving a segment, without creating an unavailable transition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100