getsentry / getsentry/sentry

Missed check-in may be skipped during partition backlogs

Open
#63,166 2 comments 0 reactions 0 assignees View on GitHub
Improvement Product Area: Crons
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
22h 21m
Merged PRs (30d)
586

Description

There exists a case where we may fail to detect missed check-ins when **any single partition** in th `ingest-monitors` topic becomes backlogged.

Two important systems must be understood to understand why this happens:

1. The `mark_missing` task is triggered by a clock that is driven by the Kafka topic. Importantly, this clock **will only move forward** once all partitions have read past the same minute. This means that if one partition is particularly slow, other partitions may consume **multiple minutes ahead**.

2. The `mark_missing` task detects missed check-ins by looking at the `MonitorEnvironment` table and querying for monitors which have their `next_checkin_latest` equal or greater than the clock tick timestamp. This value is set any time a monitor checks-in. We will compute the next expected time plus the configured margin.

What can happen is, if one partition is slow, other partitions may consume OK check-ins **AFTER** a check-in was not reported and should be considered missed.

That might look like this

```
12:00 ✅
12:01 ❌
12:02 ✅
```

If at 12:01 the monitor did not receive a check-in, but at 12:02 a check-in was received, it would compute a `next_checkin_latest` of `12:04` (The next expected is at 12:03, plus the 1 minute margin)

When the clock finally ticks forward to 12:01, we would NOT create a missed check-in for this monitor because we have already re-computed the `next_checkin_latest`.

## Potential solutions

1. Any time we consume a new check-in, we could also verify that the timestamp of our check-in was not past the `missed_margin`. If it was we can create a missed check-in, repeat until we catch up to when the check-in we just processed was expected at.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.