apache / apache/pinot

Broker routing can briefly reference segments that servers no longer serve

Open
#19,081 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Several Pinot workflows rely on broker routing converging with server-side segment state changes before the old serving path becomes unavailable. A stale routing entry can send a query to a server after that server has dropped the segment, producing SERVER_SEGMENT_MISSING.

1. Server readiness versus broker readiness
[Issue #16565](https://github.com/apache/pinot/issues/16565) describes a server returning healthy before brokers had resumed routing to it. During a rolling restart, this allowed the next server to be restarted while neither replica was queryable.

2. Segment deletion / retention
Deletion has the inverse ordering requirement. The controller updates IdealState, servers asynchronously process the resulting transition and remove the segment, and brokers asynchronously rebuild routing. If a broker routes a query using the old assignment after the server has removed the segment, the query fails.

This is less likely for ordinary retention when time pruning excludes expired segments, but it remains possible for queries spanning the retention boundary and for explicit segment deletion.

Image

3. Rebalance transition
Even with add-before-remove rebalance, a query can be dispatched immediately before the source server’s state transition reaches broker routing. If the source has removed the segment by request handling time. We have observed the following errors at least 4 times in a month due to this:
```
Query processing exceptions:
{235=1 segments [] missing on server: }
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing broker routing alongside controller IdealState updates and server-side segment state changes. Reproduce the rolling-restart, deletion, or rebalance timing described in the issue and inspect the SERVER_SEGMENT_MISSING failure. Done means routing no longer dispatches queries to servers after they stop serving a segment, without creating an unavailable transition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.