apache / apache/druid

Segment Rebalance Race

Open
#18,764 6 comments 0 reactions 1 assignee Claimed by @jtuglu1 View on GitHub
Area - Segment Balancing/Coordination Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

I've identified a race in segment movement which causes temporary segment unavailability (and thus partial query results).

Segment load/drop callbacks are racing with [prepareCurrentServers](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/PrepareBalancerAndLoadQueues.java#L139-L147).

Consider the following scenario: Coordinator is moving segment `S` from server `A` to server `B`. `S` has 1x replication.

Basically, the load/drop callbacks can happen between the start/end of this function in a way where you get a

```
SegmentReplicaCount{requiredAndLoadable=1, required=1, loaded=2, loadedNonHistorical=0, loading=0, dropping=0, movingTo=0, movingFrom=0}.
```

Essentially:
```
T0[Coordinator]: enter prepareCurrentServers()
T1[Coordinator]: Server[B] completed request[MOVE_TO] on segment[S] with status[SUCCESS]
T2[Coordinator]: Dropping segment [S] from server[A]
T3[A]: Completely removing segment[S] in [30,000]ms.
T4[Coordinator]: Server[A] completed request[DROP] on segment[S] with status[SUCCESS].
T5[Coordinator]: exit prepareCurrentServers()
T6[Coordinator]: enter prepareCluster()
T7[Coordinator]: exit prepareCluster()
T8[Coordinator]: enter initReplicaCounts()
T9[Coordinator]: exit initReplicaCounts()
T10[Coordinator]: Segment S replica count is SegmentReplicaCount{requiredAndLoadable=1, required=1, loaded=2, loadedNonHistorical=0, loading=0, dropping=0, movingTo=0, movingFrom=0}
T11[Coordinator]: Dropping segment [S] from server[B]
```

I think what's happening is that the server loop in `prepareCurrentServers()` reads the servers in a state where LOAD has persisted in the server view (2x loaded) but the DROP has not materialized yet in the view. This causes `loaded=2`. Then, I thought the in-flight DROP (since it hasn't materialized in the view) would get picked up in the `queuedSegments` load queue peons (and show up as `dropping=1`), but I think since the DROP callback returns – and clears the entry from the old `queuedSegments` load queue peons – before `prepareCluster()` has a chance to copy over the queued action to new `queuedSegments`, we lose that important bit of information. Hence, you are left in a weird state with a "valid" queue but an invalid load state. In other words, I think we need to somehow synchronize callbacks with this `prepareCurrentServers()` and `prepareCluster()`.

### Affected Version

All recent Druid versions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.