cockroachdb / cockroachdb/cockroach
kvserver: circuit breaker stuck tripped after network partition recovery
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
During chaos testing on the DRT cluster, a small number of circuit breaker replicas (3 out of ~36) were observed remaining permanently tripped after a full network partition healed. The stuck replicas were on ranges `r30197`,
`r437997`, and `r447288`. The stuck breakers only cleared 3+ hours later when a second chaos event forced a node restart.
## Impact
- `kv.closed_timestamp.max_behind_nanos` grows unboundedly (1 second per second) on the affected ranges follower reads become arbitrarily stale.
- The metric never self-heals without external intervention (restart or a
client touching the range).
## Root Cause Analysis
> RCA below is as per Claude (AI-assisted analysis)
The bug is in `refreshProposalsLocked` at `replica_raft.go:1702`.
There is a **short-circuit evaluation bug** that prevents the circuit breaker probe from being re-triggered after a snapshot clears all pending proposals.
### Problematic Code
```go
// replica_raft.go:1702
if maxSlowProposalDuration > 0 && r.breaker.Signal().Err() == nil && !destroyed {
r.breaker.TripAsync(err)
}
```
The comment on lines 1698–1701 explicitly acknowledges that calling `Signal().Err()` is critical for re-triggering the probe:
> _"NB: the call to `Err()` here also re-triggers the probe if the breaker is
> already tripped and no probe is running"_
However, due to **Go's short-circuit evaluation**, `Signal().Err()` is **never called** when `maxSlowProposalDuration == 0` (i.e., when there are no pending slow proposals after a snapshot clears the proposals map).
Datadog [link](https://us5.datadoghq.com/metric/explorer?graph_layout=multi&start=1773970200000&end=1773984600000&paused=true#N4Ig7glgJg5gpgFxALlAGwIYE8D2BXJVEADxQEYAaELcqyKBAC1pEbghkcLIF8qo4AMwgA7CAgg4RKUAiwAHOChASAtnADOcAE4RNIKtrgBHPJoQaUAbVBGN8qVoD6gnNtUZCKiOq279VKY6epbINiAiGOrKQdpYAEwGIFCeGE4a+NoAxkrIIOoIulmWgWZxyh7EyFk4WQDW2jgYWYxQAEYAdHUAbh1G8mgQWWlZENl44k5tRhh1Oh0ieKpOhRDyilBO-YPDGsBZaHgaCDrIUNoIALQtGDgalxhgGjwABG1YL8CMdwg8IHygSLRPKxLBkJIpBBpDJ4bK5fKIIolECgioYKo1eqNZqtTo9DoHO5wTZqcxReQdSpTOCMUSbSIiO77Q7HU7nK43O4PJ6vd6fb7HP4AiJReGggDMENS6UyOQqiKGyNReQw3Rg1VqDSaLXaHUG3TgIk0Gg6bAwFzacE8mBOIiyWGZRxO2jOF2ujFu90ezzeHy+PyFFEBopiZSwABYpVCZbC5XkCkiksqQKr1ZitTjdRosCa8PJSY7WS72e7PdyfXz-YL-gBdKiudx4TChcIYQYYUIqXTrYkvbZDDtJBuqJsYUM6BL-IMp9udwlaKAvUnHckvSpvGl0oduEeYcdxcHCtsQQd5AByOAELwAElbtJbPC8ADKeQ327eNvcgsOSo+z5QXleACq+Y+Eo9Y7qO+4RrWVDHFgaC5KA8htogJzKFAOAwE4BwdhoQxJG4AjaFMNB5N0bZmMigxGisCjwhkgxQEkNFwE49BMMojLuG2U7JBA9iYFgdGKMorG1jwfAgIJ4gAMJSMIMAoIsaBoDwQA) for relevant metrics.
Datadog [logs](https://us5.datadoghq.com/logs?query=cluster%3Adrt-chaos-aws%20%28%22circuit%20breaker%22%20OR%20%22tripped%22%20OR%20%22replica%20unavailable%22%29&cols=host%2Cservice&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=desc&viz=stream&from_ts=1773970200000&to_ts=1773984600000&live=false) around that time.
Jira issue: CRDB-61897
Contributor guide
Research direction
Start in kvserver/replica_raft.go at refreshProposalsLocked around line 1702, and read the nearby comment about Err() re-triggering the probe. Trace the circuit-breaker path when a snapshot clears pending proposals, then verify that recovered replicas probe again and kv.closed_timestamp.max_behind_nanos stops growing without a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100