cockroachdb / cockroachdb/cockroach

`latency jump` logs spamming the cockroach.log

Open
#93,999 4 comments 0 reactions 0 assignees View on GitHub
A-kv-server C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

I received a debug zip and the log files were filled to the brim with `latency jump` log files:

```
847921 latency jump (prev avg 11.55ms, current 96.81ms)
847922 latency jump (prev avg 3.36ms, current 7.86ms)
847923 latency jump (prev avg 5.25ms, current 52.26ms)
847924 latency jump (prev avg 6.45ms, current 11.38ms)
847925 latency jump (prev avg 4.19ms, current 24.40ms)
847926 latency jump (prev avg 4.94ms, current 10.36ms)
847927 latency jump (prev avg 7.01ms, current 18.65ms)
847928 latency jump (prev avg 8.81ms, current 17.26ms)
```

Removing these from a 7.9MB log file removes about 3MB from the log file, and have not (so far) provided any value.

**To Reproduce**

This appears in the logs for certain debug zips. I'm not sure how this is enabled or if it's universal.

**Expected behavior**

This log surely does not need to be printed as much as it is. It should be more selective to print only when outlier latency jumps occur. The noise it generates creates problems with using the logs to troubleshoot issues, and if there is a latency jump that is an outlier that we should consider, we are more likely to miss it because of the spam.

**Environment:**
- v22.1.10

**Additional Context:**

I see that the codepath for this log line is `rpc/pkg/rpc/clock_offset.go:216`. Based on this I have labeled this with KV, but I'm not sure who owns this.

Please let me know if this needs more information, or reach out to me directly or here for any questions or details.

Jira issue: CRDB-22642

Epic CRDB-54646

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.