elastic / elastic/beats

Heartbeat browser monitors that hit the global/extended timeout should still emit a summary document

Open
#51,526 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Heartbeat Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

Summary

When a browser journey exceeds the global timeout and synthexec force-terminates it (did not exit before extended timeout: 15m30s), the run produces no journey/end and no summary document. As a result:

  • The run is absent from the Synthetics monitor history (the history list is built from summary docs, one per check_group).
  • Manually triggered runs stay stuck in "pending" in the UI, because the UI waits for a summary/journey-end that never arrives.
  • There is no failed-run signal at all, even though the run physically executed.

Heartbeat is the only process still alive after the kill (the Node @elastic/synthetics subprocess is SIGKILLed and cannot flush its own summary), so heartbeat should synthesize a summary event (status down) with the timeout as the error when it terminates a run on the extended timeout.

Impact

Long-running browser journeys silently disappear from history. Operators see "the monitor isn't running" when in fact it is running every interval but each run is being killed at the timeout before emitting a summary. This makes the failure invisible in the Synthetics UI and in alerting that keys off summary docs.

Reproduction

Observed on a staging cloud-ui onboarding browser monitor (30 min schedule). The journey:

  1. Prepare session — succeeded
  2. Register — succeeded
  3. Get OTP (email MFA) — succeeded
  4. Insert OTP / verify — succeeded
  5. Onboarding (creates a resource) — succeeded
  6. Wait-for-readiness step — never emits a step/end; the journey rides the global timeout here
  7. Cleanup step — never reached

All recorded steps succeed, the last event is the step/end for step 5, and then there is no journey/end and no summary — the process is killed at 15m30s. Partial step docs are still indexed (visible in the individual test-run view), but with no summary the run never shows in history.

Expected behavior

When synthexec terminates a browser run due to the global/extended timeout, heartbeat should emit a summary document for that check_group with:

  • summary.status / monitor.status = down
  • an error describing the timeout (e.g. the did not exit before extended timeout message)

so the run is recorded as a failed run in history rather than being dropped.

Notes

  • A companion improvement in elastic/synthetics (emit a summary on its own soft timeout, before the hard kill) would be nice-to-have, but the guaranteed safety net belongs in heartbeat since it owns the forced termination.
  • Relevant code: x-pack/heartbeat/monitors/browser/synthexec/ (browser monitor runner / timeout enforcement).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in x-pack/heartbeat/monitors/browser/synthexec/, focusing on the browser monitor runner and its timeout enforcement. Trace the forced-termination path for the global or extended timeout and inspect how summary events are emitted. Done means a timed-out run emits one summary with down status and the timeout error, allowing it to appear in monitor history.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.