cockroachdb / cockroachdb/cockroach
obs/ash: add anomaly detection and ASH-derived environment sampler triggers
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Summary
Add anomaly detection to the ASH sampler and use ASH-derived signals to both emit log warnings and trigger environment sampler dumps.
This is follow-up work from #164382 and #164383, whose PRs (#165093 and #165089) were scoped down to cover periodic top-N workload summary logging and writing ASH reports alongside existing goroutine dumps/CPU profiles. The anomaly detection aspects were deferred.
[Design doc](https://docs.google.com/document/d/164fCBKytBGiRDPPMezQDGGiQClVW0wJLAzCEmwb2QKo)
## Changes needed
### Anomaly detection in ASH samples
1. **Stuck goroutine detection**: detect when the same goroutine remains in the same `WorkEvent` for an extended duration and emit a warning on the OPS log channel.
2. **LOCK contention spikes**: detect high LOCK sample accumulation for a particular statement fingerprint / workload and emit a warning.
3. **Admission queue pressure**: detect spikes in ADMISSION samples indicating queue pressure and emit a warning.
### ASH-derived environment sampler triggers
Use the anomaly signals above to proactively trigger environment sampler dumps (goroutine dumps, CPU profiles), rather than only attaching ASH reports when dumps are triggered by existing signals (CPU/memory pressure):
1. Stuck goroutine in same work state for extended duration → trigger dump
2. Spike in LOCK samples for a particular workload → trigger dump
3. Spike in ADMISSION samples → trigger dump
Each trigger should include the aggregated ASH report alongside the dump, as implemented in #165089.
Jira issue: CRDB-61206
Contributor guide
Assessment
This issue has not been assessed yet.