cockroachdb / cockroachdb/cockroach

obs/ash: add anomaly detection and ASH-derived environment sampler triggers

Open
#165,094 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

Add anomaly detection to the ASH sampler and use ASH-derived signals to both emit log warnings and trigger environment sampler dumps.

This is follow-up work from #164382 and #164383, whose PRs (#165093 and #165089) were scoped down to cover periodic top-N workload summary logging and writing ASH reports alongside existing goroutine dumps/CPU profiles. The anomaly detection aspects were deferred.

[Design doc](https://docs.google.com/document/d/164fCBKytBGiRDPPMezQDGGiQClVW0wJLAzCEmwb2QKo)

## Changes needed

### Anomaly detection in ASH samples

1. **Stuck goroutine detection**: detect when the same goroutine remains in the same `WorkEvent` for an extended duration and emit a warning on the OPS log channel.
2. **LOCK contention spikes**: detect high LOCK sample accumulation for a particular statement fingerprint / workload and emit a warning.
3. **Admission queue pressure**: detect spikes in ADMISSION samples indicating queue pressure and emit a warning.

### ASH-derived environment sampler triggers

Use the anomaly signals above to proactively trigger environment sampler dumps (goroutine dumps, CPU profiles), rather than only attaching ASH reports when dumps are triggered by existing signals (CPU/memory pressure):

1. Stuck goroutine in same work state for extended duration → trigger dump
2. Spike in LOCK samples for a particular workload → trigger dump
3. Spike in ADMISSION samples → trigger dump

Each trigger should include the aggregated ASH report alongside the dump, as implemented in #165089.

Jira issue: CRDB-61206

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.