elastic / elastic/support-diagnostics

[Feature Request] Add opt-in CPU/heap trend summary from monitoring data (7-day avg/p95/p99, daily breakdown)

Open
#1,048 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
304
Forks
175
PR merge metrics
No merged PRs in 30d

Description

### Problem

The standard diagnostic captures cluster/node state at a single point in
time. When investigating a performance issue reported after the fact
(e.g. "the cluster was slow yesterday afternoon"), a point-in-time
snapshot of CPU/heap usage is often not very useful — the problem may
have already resolved by the time the diagnostic runs, and there's no
way to tell from the snapshot whether the cluster has been trending
toward trouble over the past several days or the current state is an
outlier.

`export-monitoring.sh` already extracts raw monitoring documents for
later re-import into a separate cluster for visualization, but that's a
separate manual step that's easy to forget to run, requires standing up
somewhere to view the re-imported data, and doesn't produce a quick
human-readable summary on its own.

### Proposed Solution

When `.monitoring-es-*` indices/data streams are present, optionally run
a lightweight aggregation against the last 7 days of `node_stats` and
include a summary in the archive — both a quick per-node overview and a
daily breakdown so an engineer can see *when* a trend started, not just
that one exists:

```json
{
"size": 0,
"timeout": "10s",
"query": { "range": { "timestamp": { "gte": "now-7d" } } },
"aggs": {
"by_node": {
"terms": { "field": "source_node.name", "size": 50 },
"aggs": {
"cpu_pct": { "percentiles": { "field": "node_stats.process.cpu.percent", "percents": [50, 95, 99] } },
"heap_pct": { "percentiles": { "field": "node_stats.jvm.mem.heap_used_percent", "percents": [50, 95, 99] } },
"by_day": {
"date_histogram": { "field": "timestamp", "fixed_interval": "1d" },
"aggs": {
"cpu_pct": { "percentiles": { "field": "node_stats.process.cpu.percent", "percents": [50, 95, 99] } },
"heap_pct": { "percentiles": { "field": "node_stats.jvm.mem.heap_used_percent", "percents": [50, 95, 99] } }
}
}
}
}
}
}
```

Output written as a single summary file with both levels:

```json
{
"summary": {
"node-1": { "cpu_p95": 78.2, "heap_p95": 82.1 },
"node-2": { "cpu_p95": 45.0, "heap_p95": 60.3 }
},
"daily": {
"node-1": [
{ "date": "2026-07-21", "cpu_p95": 40.1, "heap_p95": 55.2 },
{ "date": "2026-07-27", "cpu_p95": 91.3, "heap_p95": 93.5 }
]
}
}
```

The summary gives a quick "is anything bad" scan; the daily breakdown
lets you drill into when it started, without needing a second tool.

### Architecture note (would appreciate feedback here specifically)

Every existing REST call in `elastic-rest.yml` is a plain GET URL string
keyed by version — this feature needs a `_search` request with a JSON
aggs body, which that format can't express. This would need a small
dedicated `Command` class using `RestClient.execPost` (already used
internally by `MonitoringImportProcessor` for bulk re-import, so this
wouldn't be the first POST call in the codebase, just the first one
wired into the main collection chain rather than the separate
import/export tools). Flagging this upfront since it's a departure from
the usual "add a line to `*-rest.yml`" contribution pattern, in case
there's a preferred way to structure this.

### On cluster load

Gated behind an opt-in flag (`--includeTrends`, default off) — for
self-monitored clusters, the monitoring data lives on the same cluster
that may already be struggling, so this shouldn't run without being
explicitly asked for.

To keep the query itself cheap and bounded:
- `terms` capped at a reasonable size (50) rather than open-ended.
- A 10s server-side `timeout` in the request body — if monitoring data
volume is larger than expected, the query returns partial results (or
fails) rather than running indefinitely, and the rest of the
diagnostic proceeds regardless (same `retry`/`showErrors: false`
graceful-failure pattern already used for other calls in
`elastic-rest.yml`).
- No special aggregation tuning (e.g. reduced tdigest compression) is
applied. At typical monitoring collection intervals (10s by default),
a 7-day window even across many nodes is at most a few hundred
thousand documents — well within normal `percentiles` aggregation
cost — so added tuning complexity didn't seem justified.
- Considered ES|QL as a lower-load alternative to DSL aggs; ruled it out
since its `PERCENTILE` function uses the same TDigest algorithm under
the hood (no load reduction), and it only reached GA in 8.14, while
this tool supports much older Elasticsearch versions.

Deliberately left out of this first pass, to keep the change reviewable:
- Async search / polling for the query.
- A configurable trend window or bucket interval (fixed at 7 days /
1-day buckets for now).
- Metrics beyond CPU/heap — thread pool rejections, circuit breaker
trips, GC pause time, and disk usage all looked like reasonable
next candidates while designing this, but adding them all at once
makes this harder to review. Happy to open follow-up issues for these
if the initial approach is welcome.

### Verification plan

Will verify against a running cluster with monitoring enabled: confirm
the query returns expected percentiles/daily buckets, confirm graceful
skip when `.monitoring-es-*` doesn't exist, and confirm the 10s timeout
doesn't hang the overall diagnostic run if monitoring data is unexpectedly
large.

Wanted to check direction — especially the architecture point above —
before starting on a PR.

Contributor guide

Open the contributing guide

Research direction

Start with elastic-rest.yml and the existing MonitoringImportProcessor use of RestClient.execPost; trace how REST calls enter the main collection chain and how the --includeTrends flag would be handled. Done means an opt-in 7-day CPU/heap summary with daily buckets, graceful skipping or failure, and verification against the stated running-cluster scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.