Missed check-in may be skipped during partition backlogs
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 22h 21m
- Merged PRs (30d)
- 586
Description
There exists a case where we may fail to detect missed check-ins when **any single partition** in th `ingest-monitors` topic becomes backlogged.
Two important systems must be understood to understand why this happens:
1. The `mark_missing` task is triggered by a clock that is driven by the Kafka topic. Importantly, this clock **will only move forward** once all partitions have read past the same minute. This means that if one partition is particularly slow, other partitions may consume **multiple minutes ahead**.
2. The `mark_missing` task detects missed check-ins by looking at the `MonitorEnvironment` table and querying for monitors which have their `next_checkin_latest` equal or greater than the clock tick timestamp. This value is set any time a monitor checks-in. We will compute the next expected time plus the configured margin.
What can happen is, if one partition is slow, other partitions may consume OK check-ins **AFTER** a check-in was not reported and should be considered missed.
That might look like this
```
12:00 ✅
12:01 ❌
12:02 ✅
```
If at 12:01 the monitor did not receive a check-in, but at 12:02 a check-in was received, it would compute a `next_checkin_latest` of `12:04` (The next expected is at 12:03, plus the 1 minute margin)
When the clock finally ticks forward to 12:01, we would NOT create a missed check-in for this monitor because we have already re-computed the `next_checkin_latest`.
## Potential solutions
1. Any time we consume a new check-in, we could also verify that the timestamp of our check-in was not past the `missed_margin`. If it was we can create a missed check-in, repeat until we catch up to when the check-in we just processed was expected at.
Contributor guide
Assessment
This issue has not been assessed yet.