cockroachdb / cockroachdb/cockroach

kv: noisy log entries when cluster is overloaded

Open
#118,655 1 comment 0 reactions 0 assignees View on GitHub
A-kv-replication C-bug O-testcluster P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We're seeing a couple of log messages on the DRT cluster which occur quite frequently. In an ~3 hour period, we're seeing over 20K occurrences of these two messages:

```
I240131 12:13:11.522180 2147 kv/kvserver/store_raft.go:699 ⋮ [T1,Vsystem,n13,s13,r251625/4:‹/Tenant/3/Table/33156/1/5…›,raft] 220683 raft ready handling: 0.52s [append=0.00s, apply=0.00s, non-blocking-sync=0.52s, other=0.00s], wrote [append-batch=291 B, append-ent=224 B (1), ]; node might be overloaded
W240131 12:13:11.272743 2248 2@rpc/clock_offset.go:291 ⋮ [T1,Vsystem,n13,rnode=8,raddr=‹10.142.0.158:26257›,class=raft,rpc] 220667 latency jump (prev avg 24.30ms, current 160.45ms)
```

As observed on n13 of the DRT cluster:

```
ubuntu@cct-232-0013$ grep "node might be overloaded" cockroach.cct-232-0013.ubuntu.2024-01-31T12_13_11Z.1088640.log cockroach.cct-232-0013.ubuntu.2024-01-31T13_15_19Z.1088640.log | wc -l
22615
ubuntu@cct-232-0013$ grep "latency jump" cockroach.cct-232-0013.ubuntu.2024-01-31T12_13_11Z.1088640.log cockroach.cct-232-0013.ubuntu.2024-01-31T13_15_19Z.1088640.log | wc -l
22004
```

Does it make sense to wrap these two messages in a `log.everyN`?

Jira issue: CRDB-35822

Epic CRDB-55046

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.