cockroachdb / cockroachdb/cockroach

server: goroutine dumper should reset maxGoroutines periodically

Open
#114,836 0 comments 1 reaction 0 assignees View on GitHub
A-observability-inf C-bug O-support P-3 T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

As part of the runtime stats sampler task, the goroutine dumper will record and dump goroutines if we meet all of the following conditions:
1. numGoroutines > `server.goroutine_dump.num_goroutines_threshold`
2. numGoroutines >= `maxGoroutinesDumpedSoFar`

Initially, we just need 1 to be true and then on each dump we record the new value of `maxGoroutinesDumpedSoFar`. This is presumably so that we don't dump unnecessarily. We can naturally get to what is considered a high number of goroutines for that specific process when it reaches a steadier state, and only dump when we reach that high value.

The problem is the running max is only reset when the initial threshold (`server.goroutine_dump.num_goroutines_threshold`) is changed. This means if the equilibrium number of goroutines goes down or if we record an abnormally high number of goroutines just once, it's likely we will never record (or miss important) goroutine dumps moving forward.

To reduce the risk of missing important goroutine dumps we can periodically reset the `maxGoroutines` recorded to ensure it's set back to a reasonable value for the current state.

Jira issue: CRDB-33706

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.