cockroachdb / cockroachdb/cockroach

Improve Goroutine Collection

Open
#144,860 0 comments 0 reactions 1 assignee Claimed by @kevin-v-ngo View on GitHub
C-enhancement O-support P-3 T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Goroutine dumps are often required in order to perform Root Cause Analysis in CockroachDB. These dumps are especially useful while an issue is occuring so you can see what's happening.

Part of incident resolution could be restarting a node or all nodes. Operations such as upgrading a cluster, or anything that can trigger a restart of a node will wipe the active goroutines.

**Describe the solution you'd like**

The ability to persist the groutines prior to restarting the node. The reason this solution would work is because stopping to pull goroutines during an active incident where the resolution is mitigation is to restart the node is not worth the time spent. The goal is to mitigate the issue, and focus on RCA after. The challenge is the mitigation erases the active goroutines.

One solution would be to dump the goroutines before any planned restart.

**Describe alternatives you've considered**

Folks would have to remember to pull it before mitigation, and that's not a cheap or simple consideration.

**Additional context**

The proposed solution may not work for serverless because the log directories may be ephemeral.

Another consideration would be to have the SU dashboard pull these dumps directly into Datadog for analysis using the existing button that pulls non-log files in.

Jira issue: CRDB-49574

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.