apache / apache/druid

Segment Load/Drop Race Causes Silent Partial Query Results

Open
#18,738 3 comments 0 reactions 0 assignees View on GitHub
Area - Querying Bug Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

All recent Druid versions.

### Description

Using HTTP-based load/drop segment communication between data nodes and brokers:
T0: Coordinator moves segment S from historical A to historical B.
T1: Historical B loads segment S
T2: Coordinator gets loadSegment callback from B
T3: Coordinator tells Historical A to drop segment S
T4: Broker gets drop callback from Historical A, removes the server from the timeline (now 0 nodes serving)
T5: Query sees partial results/no servers for that segment
T6: Broker gets load callback from Historical B

This out-of-order delivery causes a momentary lapse in the Broker's timeline, yielding partial query results to the user without them knowing. In large deployments, this occurs quite frequently – nearly every hour.

Proposal: Brokers should rely on Coordinators for timeline add/drop callbacks, instead of talking to data nodes directly. While placing more responsibility on the coordinator, this ensures happens-before ordering is respected and there's a single, linearizable chain of events seen by each broker.

https://github.com/apache/druid/issues/18716 will fail these queries instead of letting them succeed, but the queries will still unnecessarily fail which is not ideal.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Coordinator, broker, and data-node segment load/drop callback paths described in the issue, and read the related failure behavior in issue #18716. Determine how callback ordering can be made linearizable from the Coordinator to brokers. Done means segment moves do not create a momentary empty broker timeline or silently return partial results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.