cockroachdb / cockroachdb/cockroach

kvserver: circuit breaker stuck tripped after network partition recovery

Open
#166,391 0 comments 0 reactions 0 assignees View on GitHub
branch-master C-bug O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

During chaos testing on the DRT cluster, a small number of circuit breaker replicas (3 out of ~36) were observed remaining permanently tripped after a full network partition healed. The stuck replicas were on ranges `r30197`,
`r437997`, and `r447288`. The stuck breakers only cleared 3+ hours later when a second chaos event forced a node restart.

## Impact
- `kv.closed_timestamp.max_behind_nanos` grows unboundedly (1 second per second) on the affected ranges follower reads become arbitrarily stale.
- The metric never self-heals without external intervention (restart or a
client touching the range).
Image

## Root Cause Analysis

> RCA below is as per Claude (AI-assisted analysis)

The bug is in `refreshProposalsLocked` at `replica_raft.go:1702`.

There is a **short-circuit evaluation bug** that prevents the circuit breaker probe from being re-triggered after a snapshot clears all pending proposals.

### Problematic Code

```go
// replica_raft.go:1702
if maxSlowProposalDuration > 0 && r.breaker.Signal().Err() == nil && !destroyed {
r.breaker.TripAsync(err)
}
```

The comment on lines 1698–1701 explicitly acknowledges that calling `Signal().Err()` is critical for re-triggering the probe:
> _"NB: the call to `Err()` here also re-triggers the probe if the breaker is
> already tripped and no probe is running"_

However, due to **Go's short-circuit evaluation**, `Signal().Err()` is **never called** when `maxSlowProposalDuration == 0` (i.e., when there are no pending slow proposals after a snapshot clears the proposals map).

Datadog [link](https://us5.datadoghq.com/metric/explorer?graph_layout=multi&start=1773970200000&end=1773984600000&paused=true#N4Ig7glgJg5gpgFxALlAGwIYE8D2BXJVEADxQEYAaELcqyKBAC1pEbghkcLIF8qo4AMwgA7CAgg4RKUAiwAHOChASAtnADOcAE4RNIKtrgBHPJoQaUAbVBGN8qVoD6gnNtUZCKiOq279VKY6epbINiAiGOrKQdpYAEwGIFCeGE4a+NoAxkrIIOoIulmWgWZxyh7EyFk4WQDW2jgYWYxQAEYAdHUAbh1G8mgQWWlZENl44k5tRhh1Oh0ieKpOhRDyilBO-YPDGsBZaHgaCDrIUNoIALQtGDgalxhgGjwABG1YL8CMdwg8IHygSLRPKxLBkJIpBBpDJ4bK5fKIIolECgioYKo1eqNZqtTo9DoHO5wTZqcxReQdSpTOCMUSbSIiO77Q7HU7nK43O4PJ6vd6fb7HP4AiJReGggDMENS6UyOQqiKGyNReQw3Rg1VqDSaLXaHUG3TgIk0Gg6bAwFzacE8mBOIiyWGZRxO2jOF2ujFu90ezzeHy+PyFFEBopiZSwABYpVCZbC5XkCkiksqQKr1ZitTjdRosCa8PJSY7WS72e7PdyfXz-YL-gBdKiudx4TChcIYQYYUIqXTrYkvbZDDtJBuqJsYUM6BL-IMp9udwlaKAvUnHckvSpvGl0oduEeYcdxcHCtsQQd5AByOAELwAElbtJbPC8ADKeQ327eNvcgsOSo+z5QXleACq+Y+Eo9Y7qO+4RrWVDHFgaC5KA8htogJzKFAOAwE4BwdhoQxJG4AjaFMNB5N0bZmMigxGisCjwhkgxQEkNFwE49BMMojLuG2U7JBA9iYFgdGKMorG1jwfAgIJ4gAMJSMIMAoIsaBoDwQA) for relevant metrics.
Datadog [logs](https://us5.datadoghq.com/logs?query=cluster%3Adrt-chaos-aws%20%28%22circuit%20breaker%22%20OR%20%22tripped%22%20OR%20%22replica%20unavailable%22%29&cols=host%2Cservice&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=desc&viz=stream&from_ts=1773970200000&to_ts=1773984600000&live=false) around that time.

Jira issue: CRDB-61897

Contributor guide

Open the contributing guide

Research direction

Start in kvserver/replica_raft.go at refreshProposalsLocked around line 1702, and read the nearby comment about Err() re-triggering the probe. Trace the circuit-breaker path when a snapshot clears pending proposals, then verify that recovered replicas probe again and kv.closed_timestamp.max_behind_nanos stops growing without a restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.