Segment Load/Drop Race Causes Silent Partial Query Results
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
All recent Druid versions.
### Description
Using HTTP-based load/drop segment communication between data nodes and brokers:
T0: Coordinator moves segment S from historical A to historical B.
T1: Historical B loads segment S
T2: Coordinator gets loadSegment callback from B
T3: Coordinator tells Historical A to drop segment S
T4: Broker gets drop callback from Historical A, removes the server from the timeline (now 0 nodes serving)
T5: Query sees partial results/no servers for that segment
T6: Broker gets load callback from Historical B
This out-of-order delivery causes a momentary lapse in the Broker's timeline, yielding partial query results to the user without them knowing. In large deployments, this occurs quite frequently – nearly every hour.
Proposal: Brokers should rely on Coordinators for timeline add/drop callbacks, instead of talking to data nodes directly. While placing more responsibility on the coordinator, this ensures happens-before ordering is respected and there's a single, linearizable chain of events seen by each broker.
https://github.com/apache/druid/issues/18716 will fail these queries instead of letting them succeed, but the queries will still unnecessarily fail which is not ideal.
Contributor guide
Research direction
Start by tracing the Coordinator, broker, and data-node segment load/drop callback paths described in the issue, and read the related failure behavior in issue #18716. Determine how callback ordering can be made linearizable from the Coordinator to brokers. Done means segment moves do not create a momentary empty broker timeline or silently return partial results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100