cockroachdb / cockroachdb/cockroach
cli/zip,obs/ash: generate a single aggregated ASH report covering the full debug.zip period
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Today, the only Active Session History (ASH) data that lands in a `debug.zip` is the opportunistic `ash_report...{txt,json}` files written under `nodes//heapprof/`. These have two limitations that make them hard to use for triage:
1. **They only exist if a CPU profile or goroutine dump happened to fire** — they're a side effect of the env sampler's threshold-based triggers, not something produced for the zip itself. A zip taken on a healthy-looking node may contain no ASH report at all.
2. **Each report only covers `obs.ash.log_interval` (default 10m)** of history, and that window is coupled to the OPS-channel log-summary cadence (can't be widened independently). A single 10-minute snapshot doesn't show how the workload profile evolved over the period the zip is meant to explain.
Meanwhile the full historical data *is* persisted in `system.active_session_history`, but that table is **deliberately excluded** from debug.zip because the raw rows are too large to download ([`zip_table_registry.go:1289`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/cli/zip_table_registry.go#L1263-L1289)). So operators are left with either nothing or a fragmentary 10-minute view.
This was raised by the field: a 10-minute sample isn't useful, and there's a desire for ASH data covering the entire debug.zip period.
**Describe the solution you'd like**
At debug.zip collection time, generate **one aggregated ASH report** covering the full relevant period (or a configurable lookback), independent of the profiler triggers. Because it is *aggregated* rather than raw, it stays compact and sidesteps the "too much to download" concern that excludes `system.active_session_history`.
Suggested shape:
- Source from the persisted `system.active_session_history` ([`ashflush/flusher.go`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/obs/ash/ashflush/flusher.go)) so coverage isn't bounded by the in-memory ring buffer, aggregating server-side.
- **Time-bucket** the aggregation (e.g. per-N-minute buckets) rather than one flat roll-up, so the report shows how the top workloads/wait events changed across the period — directly addressing the "one snapshot isn't enough" complaint.
- Reuse the existing aggregation/report writers ([`AggregateSamples`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/obs/ash/aggregate.go#L30), [`WriteTextReport`/`WriteJSONReport`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/obs/ash/report.go#L26)) for output format.
- Tie the window to the zip's `--files-from`/`--files-until` range when provided, otherwise a sensible default (e.g. last hour) with an override.
**Describe alternatives you've considered**
- Including the raw `system.active_session_history` in debug.zip — rejected upstream as too large.
- Simply widening `obs.ash.log_interval` — partially helps the existing opportunistic reports, but they still only appear when a profile/dump fires, and a single flat window loses time-evolution. (Decoupling that setting is worth tracking separately.)
**Additional context**
Related: #172147 (empty ASH report when ASH is disabled). Both are about making ASH in debug.zip more useful and self-explanatory for triage.
Jira issue: CRDB-65292
Contributor guide
Research direction
Start with the debug.zip collection path and pkg/cli/zip_table_registry.go, then read pkg/obs/ash/ashflush/flusher.go, aggregate.go, and report.go. Determine how the --files-from/--files-until window and default lookback should reach the persisted ASH query; done means one compact, time-bucketed report is produced at zip collection independently of profiler triggers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- cli, databases, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100