cockroachdb / cockroachdb/cockroach
server/profiler: ASH report is written as all-zeros when ASH is disabled, with no indication why
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
When a CPU profile or goroutine dump is taken by the env sampler, an ASH report is written alongside it under `nodes//heapprof/` in `debug.zip` (`ash_report...{txt,json}`). However, when ASH sampling is disabled (`obs.ash.enabled = false`, which is the default in 26.2), the report is still written — as an all-zeros document:
```json
{ "generated": "...", "lookback_seconds": 600, "total_samples": 0, "groups": [] }
```
This is confusing during debug.zip triage: an operator who finds an empty report cannot tell whether (a) ASH was disabled, (b) there was genuinely no activity in the lookback window, or (c) something is broken. This came up in a support-style investigation where empty reports were initially suspected to be a bug, when in fact ASH was simply off.
**Describe the solution you'd like**
Make the empty report self-explanatory. Either:
1. Skip writing the report entirely when `obs.ash.enabled = false`, or
2. Include an explicit `"ash_enabled": false` field (and an equivalent line in the text report) so the reason for the empty report is unambiguous.
Note: a `sampler == nil` check is insufficient — the global sampler is initialized regardless of `obs.ash.enabled`; it just doesn't accumulate samples while disabled. The check should read `ash.Enabled.Get(&st.SV)`.
**Describe alternatives you've considered**
Leaving as-is and relying on operators to know ASH defaults off. This is what caused the confusion in the first place.
**Additional context**
- Report is written here regardless of enabled state: [`ashreportprofiler.go:78`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/server/profiler/ashreportprofiler.go#L78)
- Triggered from the env sampler on CPU profile / goroutine dump: [`env_sampler.go:280`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/server/env_sampler.go#L280-L298)
- Verified against v26.2.1: with ASH disabled the report is all-zeros; with `obs.ash.enabled=true` the same trigger produces populated reports.
Jira issue: CRDB-65291
Contributor guide
Research direction
Read pkg/server/profiler/ashreportprofiler.go around line 78, then trace the env-sampler call in pkg/server/env_sampler.go around lines 280-298. Check the ASH enabled state with ash.Enabled.Get(&st.SV) and verify that a disabled sampler no longer produces an unexplained all-zero report, or that both JSON and text reports clearly identify ASH as disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- Half a day
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100