cockroachdb / cockroachdb/cockroach

problemranges: not showing tripped replicas

Open
#101,999 0 comments 0 reactions 0 assignees View on GitHub
C-bug O-testcluster T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Via 23.1 testcluster at v23.1.0-beta.2. These two screenshots below should be mutually exclusive. I think the metric is right; n8 has >300 replicas for which the circuit breaker is tripped.

image

image

**To Reproduce**

Unsure. The history of the testcluster is somewhat interesting and could be relevant.
The tripped replicas are in all likelihood leftover replicas of merges which haven't seen their merge yet. They are unable to make forward progress, so their breaker trips. But they're basically replicas for ranges that "no longer exist" (though the replicas very much exist).
Since there is a memory leak in this situation, we disabled circuit breakers. This did not untrip the existing tripped replicas. This too might have to do with the bug.
But the likeliest explanation is still that the problem ranges page is broken in a simpler way, nothing I describe above should have an obvious way in which it would confuse the page.

**Expected behavior**

See 304 tripped replicas for n8 in problem ranges.

Jira issue: CRDB-27208

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.