elastic / elastic/integrations
[elastic_agent] Bump time window on agent status alerting rule templates to mitigate false recoveries
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Summary
The OOTB agent status alerting rule templates (`elastic-agent-unhealthy-status`, `elastic-agent-offline-status`, `elastic-agent-unenrolled-status`, `elastic-agent-uninstalled-status`) recover incorrectly while the underlying condition still holds.
Rules: https://github.com/elastic/integrations/tree/main/packages/elastic_agent/kibana/alerting_rule_template
This will be best solved with alerting_v2 when a `recovery_condition` can defined separately for ESQL rules. Separate issue for addressing this from that angle: https://github.com/elastic/integrations/issues/18886#issue-4402449223
For now we'll need to find a solution within the current rule behavior.
## Problem
These rules query `logs-elastic_agent.status_change-default` over a 5-minute window. The data stream is intentionally populated only on **transitions**, so an agent that is "stuck unhealthy" emits no further docs. After the originating transition rolls out of the 5m window, the rule sees zero hits and falsely recovers even though the agent is still unhealthy. With `excludeHitsFromPreviousRun: true`, this happens within ~1 minute.
## Suggestions
Change each query to evaluate the **latest** status_change per agent, and group results so each agent gets its own alert instance. Example:
```
FROM logs-elastic_agent.status_change-default
| WHERE data_stream.dataset == "elastic_agent.status_change" AND agentless == false
| STATS latest_health_status = LAST(health_status, @timestamp) BY agent.id
| WHERE latest_health_status == ""
```
Rule param changes:
- groupBy: top, termField: agent.id per-agent alert instances; recovery happens for an agent the moment its latest transition is no longer in the alerting state.
- excludeHitsFromPreviousRun: false — required for LAST() to operate on the full windowed set.
- timeWindowSize: bump from 5m to a value chosen by benchmarking (see below). This window functions as a means to keep alerting on the unhealthy status but incidentally also becomes a TTL for the alert: an agent that has been stuck in an alerting state with no further transitions for longer than the window will not have visible alerts.
## Acceptance criteria
- All elastic agent status rule templates updated to evaluate
- LAST() status change event by agent.id
- This should result in a unique alert per agent when it goes unhealthy
- Window size chosen through benchmarking. Test rule execution latency stays acceptable across realistic fleet sizes and worst case status_change document volume.
- Rule execution history captures query timing.
- Example query window sizes to consider: 6h, 24h, 3d, 7d.
- We'll have to balance here query performance and a reasonable time for users to notice the alert being active.
- When an agent truely recovers within this window, the alert should also recover
- The chosen window is documented as an effective TTL on the alert in the integration's README/changelog
- Public docs on monitoring agent health should reflect this expectation
Contributor guide
Assessment
This issue has not been assessed yet.