bluewave-labs / bluewave-labs/Checkmate
Mitigating false incidents temporary egress failiure
- Dominant language
- TypeScript
- Stars
- 10.8k
- Forks
- 1.2k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 68
Description
**Is your feature request related to a problem? Please describe.**
When our server or cluster loses outbound internet connectivity for a few minutes, every monitor's checks fail at the same time, since they all run from the same egress path. Checkmate has no way to distinguish this from a genuine outage of the monitored targets, so it fires incidents and alerts for every monitor at once, even though in reality nothing may actually be down.
**Describe the solution you'd like**
An instance-level "egress self-check": in Settings, let an admin specify a small list of known-reliable domains or IPs to probe on a schedule (e.g. 1.1.1.1, 8.8.8.8, or a custom list). If all of them fail, treat this as evidence that the instance itself has lost connectivity, and suppress incident creation and alerting until the check succeeds again. Include an option to send a notification when this state is detected, using the same notification channels (email, Slack, Discord, webhook, etc.) already configurable for monitors.
**Describe alternatives you've considered**
- Raising `statusWindowSize`/`statusWindowThreshold` on monitors to require more failures before flipping to down. Reduces false positives somewhat but delays real-incident detection too, and doesn't address the root cause.
- Routing monitors through a proxy on a separate network path (`proxyMode: custom`), so their checks don't share the instance's own egress. Effective per monitor, but requires hosting a proxy elsewhere and doesn't help monitors left on the default path.
- Using the existing Geo-Distributed Checks (Globalping) data as a cross-check against a monitor's own failed check before creating an incident. More precise, but only covers monitors with geo-checks enabled (HTTP/ping only), and ties the incident-decision path to a third-party API rather than just a display feature.
**Additional context**
Happy to help scope or implement this — I'd been exploring how it could hook into the existing `AppSettings` singleton and notification pipeline. Would like to know if this sounds useful to others too.
Contributor guide
Research direction
Start by tracing the existing AppSettings singleton and notification pipeline mentioned in the issue, then inspect how monitor failures create incidents and alerts. Define the egress-check configuration, failure state, recovery behavior, and notification flow before implementation; done means temporary shared connectivity loss no longer creates monitor incidents and the configured state notification works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100