koala73 / koala73/worldmonitor

fix(analytics): collector env-failure aggregate floor still latching ~340 windows/day after the #5974 tuning round

Open
#6,332 0 comments 0 reactions 0 assignees View on GitHub
area: infrastructure bug P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Summary

The cross-user collector health alarm (`api/analytics-health.js` → "Umami collector environment failures crossed aggregate floor") has been latching ~340 windows/day continuously since 2026-08-01 18:43Z, through 2026-08-08 03:57Z (latest triage read). This is AFTER the #5973→#5974 "alert only on actionable failures" round (merged 2026-08-02) and the #6053/#6057 pool-wait bounding round. Either the collector still fails a large share of minutes, or the aggregate floor still passes environmental noise — both need an owner.

Priority suggestion: P2 (observability is blinded either way: at this volume the alarm is background noise, so a real outage would not stand out).

## Evidence (Sentry, 2026-08-01 → 2026-08-08)

Server-side aggregate alarm — fires at most once per 60 s bucket per cohort (Redis `SET NX` latch, `api/analytics-health.js:82-88`), so event count ≈ distinct majority-failure windows:

| Sentry issue | cohort | events | users | first seen | last seen |
|---|---|---|---|---|---|
| WORLDMONITOR-Y7 (7646866216) | event | 1,227 | 887 | 08-01 18:44Z | 08-08 03:57Z |
| WORLDMONITOR-Y6 (7646865720) | identify | 1,128 | 756 | 08-01 18:43Z | 08-08 03:52Z |

≈2,355 latched windows in 6.4 days ≈ 1 in every 4 minutes has a 60 s cross-user bucket with ≥5 writes and ≥50% failures (floor: `MIN_WRITES = 5`, `MIN_FAILURE_RATE = 0.5`, `api/analytics-health.js:21-22`). Sampled event extras show `writeCount` 5–8, `failureRate` 0.5–0.6 — the floor is being crossed at the minimum sample size, not during big bursts.

Client-side captures in the same window (`Umami collector write failed`, per-`failureKind` fingerprints):

| Sentry issue | failureKind | events | users | window |
|---|---|---|---|---|
| WORLDMONITOR-YC (7651107112) | network | 43 | 23 | 08-04 → 08-07 |
| WORLDMONITOR-YD (7651157469) | queue-overflow | 30 | 30 | 08-04 → 08-07 (evidence posted on #6288) |
| WORLDMONITOR-YF (7651226146) | timeout | 16 | 14 | 08-04 → 08-08 |
| WORLDMONITOR-YE (7651197639) | missing-receipt | 10 | 10 | 08-04 → 08-07 |
| WORLDMONITOR-YK (7652554919) | http | 63 | 61 | 08-04 20:28 → 21:41 only (burst, self-resolved) |

## The fork to resolve

1. **Collector still degraded**: mixed kinds (timeout + missing-receipt + http burst) alongside the aggregate floor suggests real write-path failures continuing after the August fixes. Cross-check Railway collector logs / #6024-style memory pressure for 08-01 → 08-08, especially the 08-04 20:28–21:41Z http burst.
2. **Floor mis-sized for low-traffic minutes**: `MIN_WRITES = 5` in a 60 s cross-user bucket is small at off-peak; a handful of ad-blocker/privacy-layer users can own an entire bucket. If (1) comes back clean, the floor needs a higher minimum sample or a sustained-windows requirement before latching.

## Acceptance criteria

- [ ] Attribute the 08-01 → 08-08 alarm volume: real write failures vs environmental noise, with collector-side evidence (not client-side counters alone).
- [ ] Either fix the collector failure mode or resize the floor (`MIN_WRITES` / `MIN_FAILURE_RATE` / sustained-window requirement) so the alarm only fires on actionable degradation.
- [ ] After the change, the alarm rate drops to a level where a single latched window is worth a human look; verify over ≥48 h of production.
- [ ] Existing Sentry issues Y7/Y6 stop accumulating at ~340/day (they are left unresolved as live monitors until then).

## Related

- #6288 (open) — client transport queue-parking; owns the `queue-overflow` slice.
- #5974 (merged 08-02) — previous floor-tuning round; this issue is the evidence that tuning did not converge.
- #6024 / #6053 / #6057 — August collector OOM/pool saga.

Contributor guide

Open the contributing guide

Research direction

Start with api/analytics-health.js, especially MIN_WRITES, MIN_FAILURE_RATE, and the Redis SET NX latch at lines 82-88. Compare the 08-01 to 08-08 Sentry windows with Railway collector logs, including the 08-04 HTTP burst and memory-pressure history. Done means the failure source is attributed, the collector or alarm floor is corrected, and production monitoring shows actionable rates for at least 48 hours.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis, typescript
Domain
backend, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.