getsentry / getsentry/sentry

Metric Monitors: Detect ingestion delays

Open
#104,860 2 comments 0 reactions 0 assignees View on GitHub
1. Urgent Feature Product Area: Alerts
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
22h 21m
Merged PRs (30d)
586

Description

### Problem Statement

Similar to https://github.com/getsentry/sentry/issues/79328

Challenge:

> Throughput monitoring works well for crons to duck alerting because we have a well known traffic pattern. If we suddenly start receiving less cron check-ins, we can almost be 100% positive it's an internal infrastructure problem.

This is not necessarily the case for other data sets (errors, metrics, logs, spans etc.)

### Solution Brainstorm

We could potentially use the same signals from crons to stop other occurrences however ingestion delays or outages are not always equally distributed

Similar approaches with respect to various forms of telemetry or the ability to mute moinitors/alerts during backfill periods or known issues would result in fewer false positive incidents/pages for customers that rely on these signals

### Product Area

Alerts

Contributor guide

Open the contributing guide

Research direction

Start by reading the related issue #79328 and the discussion in this issue. Clarify which telemetry types, ingestion-delay signals, and mute or backfill behaviors are in scope before identifying implementation files and tests; done should mean an agreed, testable design for reducing false alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.