cockroachdb / cockroachdb/cockroach

server: improve goroutine dumper heuristics for long-running nodes with many goroutines

Open
#103,321 0 comments 0 reactions 0 assignees View on GitHub
A-observability-inf C-enhancement O-support P-3 T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The goroutine dumper currently uses a single heuristic to decide when to capture a goroutine profile:
https://github.com/cockroachdb/cockroach/blob/1ceb218e2fbc92413344d041682b197228f183ea/pkg/server/goroutinedumper/goroutinedumper.go#L60-L66

This heuristic is insufficient in two ways:
1. it ratchets up the `maxGoroutinesDumped` without ever dropping it back down. This means that a goroutine spike early in a process' lifetime can prevent the dumper from ever capturing a profile again.
2. it only dumps when the goroutine count is double the previous count. This is effective for nodes with a small number of goroutines, but quickly means that even large spikes of goroutines (e.g. 8,000 to 15,999) no longer trigger the dumper.

We should rethink these heuristics to better ensure that when a node's goroutine count spikes, we capture a goroutine dump. Some possible ideas are:
1. capture a dump every increase in goroutines by 1000, instead of every doubling.
2. reset the `maxGoroutinesDumped` to the current goroutine count every hour.

Jira issue: CRDB-27946

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.