cockroachdb / cockroachdb/cockroach

cli/zip,obs/ash: generate a single aggregated ASH report covering the full debug.zip period

Open
#172,148 0 comments 0 reactions 0 assignees View on GitHub
A-cluster-observability C-enhancement O-agent T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Today, the only Active Session History (ASH) data that lands in a `debug.zip` is the opportunistic `ash_report...{txt,json}` files written under `nodes//heapprof/`. These have two limitations that make them hard to use for triage:

1. **They only exist if a CPU profile or goroutine dump happened to fire** — they're a side effect of the env sampler's threshold-based triggers, not something produced for the zip itself. A zip taken on a healthy-looking node may contain no ASH report at all.
2. **Each report only covers `obs.ash.log_interval` (default 10m)** of history, and that window is coupled to the OPS-channel log-summary cadence (can't be widened independently). A single 10-minute snapshot doesn't show how the workload profile evolved over the period the zip is meant to explain.

Meanwhile the full historical data *is* persisted in `system.active_session_history`, but that table is **deliberately excluded** from debug.zip because the raw rows are too large to download ([`zip_table_registry.go:1289`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/cli/zip_table_registry.go#L1263-L1289)). So operators are left with either nothing or a fragmentary 10-minute view.

This was raised by the field: a 10-minute sample isn't useful, and there's a desire for ASH data covering the entire debug.zip period.

**Describe the solution you'd like**

At debug.zip collection time, generate **one aggregated ASH report** covering the full relevant period (or a configurable lookback), independent of the profiler triggers. Because it is *aggregated* rather than raw, it stays compact and sidesteps the "too much to download" concern that excludes `system.active_session_history`.

Suggested shape:

- Source from the persisted `system.active_session_history` ([`ashflush/flusher.go`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/obs/ash/ashflush/flusher.go)) so coverage isn't bounded by the in-memory ring buffer, aggregating server-side.
- **Time-bucket** the aggregation (e.g. per-N-minute buckets) rather than one flat roll-up, so the report shows how the top workloads/wait events changed across the period — directly addressing the "one snapshot isn't enough" complaint.
- Reuse the existing aggregation/report writers ([`AggregateSamples`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/obs/ash/aggregate.go#L30), [`WriteTextReport`/`WriteJSONReport`](https://github.com/cockroachdb/cockroach/blob/c6bf8649e12010033f6fe9038884147bb4521ca7/pkg/obs/ash/report.go#L26)) for output format.
- Tie the window to the zip's `--files-from`/`--files-until` range when provided, otherwise a sensible default (e.g. last hour) with an override.

**Describe alternatives you've considered**

- Including the raw `system.active_session_history` in debug.zip — rejected upstream as too large.
- Simply widening `obs.ash.log_interval` — partially helps the existing opportunistic reports, but they still only appear when a profile/dump fires, and a single flat window loses time-evolution. (Decoupling that setting is worth tracking separately.)

**Additional context**

Related: #172147 (empty ASH report when ASH is disabled). Both are about making ASH in debug.zip more useful and self-explanatory for triage.

Jira issue: CRDB-65292

Contributor guide

Open the contributing guide

Research direction

Start with the debug.zip collection path and pkg/cli/zip_table_registry.go, then read pkg/obs/ash/ashflush/flusher.go, aggregate.go, and report.go. Determine how the --files-from/--files-until window and default lookback should reach the persisted ASH query; done means one compact, time-bucketed report is produced at zip collection independently of profiler triggers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
cli, databases, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.