cockroachdb / cockroachdb/cockroach
server: improve goroutine dumper heuristics for long-running nodes with many goroutines
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The goroutine dumper currently uses a single heuristic to decide when to capture a goroutine profile:
https://github.com/cockroachdb/cockroach/blob/1ceb218e2fbc92413344d041682b197228f183ea/pkg/server/goroutinedumper/goroutinedumper.go#L60-L66
This heuristic is insufficient in two ways:
1. it ratchets up the `maxGoroutinesDumped` without ever dropping it back down. This means that a goroutine spike early in a process' lifetime can prevent the dumper from ever capturing a profile again.
2. it only dumps when the goroutine count is double the previous count. This is effective for nodes with a small number of goroutines, but quickly means that even large spikes of goroutines (e.g. 8,000 to 15,999) no longer trigger the dumper.
We should rethink these heuristics to better ensure that when a node's goroutine count spikes, we capture a goroutine dump. Some possible ideas are:
1. capture a dump every increase in goroutines by 1000, instead of every doubling.
2. reset the `maxGoroutinesDumped` to the current goroutine count every hour.
Jira issue: CRDB-27946
Contributor guide
Assessment
This issue has not been assessed yet.