koala73 / koala73/worldmonitor
fix(analytics): collector env-failure aggregate floor still latching ~340 windows/day after the #5974 tuning round
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Summary
The cross-user collector health alarm (`api/analytics-health.js` → "Umami collector environment failures crossed aggregate floor") has been latching ~340 windows/day continuously since 2026-08-01 18:43Z, through 2026-08-08 03:57Z (latest triage read). This is AFTER the #5973→#5974 "alert only on actionable failures" round (merged 2026-08-02) and the #6053/#6057 pool-wait bounding round. Either the collector still fails a large share of minutes, or the aggregate floor still passes environmental noise — both need an owner.
Priority suggestion: P2 (observability is blinded either way: at this volume the alarm is background noise, so a real outage would not stand out).
## Evidence (Sentry, 2026-08-01 → 2026-08-08)
Server-side aggregate alarm — fires at most once per 60 s bucket per cohort (Redis `SET NX` latch, `api/analytics-health.js:82-88`), so event count ≈ distinct majority-failure windows:
| Sentry issue | cohort | events | users | first seen | last seen |
|---|---|---|---|---|---|
| WORLDMONITOR-Y7 (7646866216) | event | 1,227 | 887 | 08-01 18:44Z | 08-08 03:57Z |
| WORLDMONITOR-Y6 (7646865720) | identify | 1,128 | 756 | 08-01 18:43Z | 08-08 03:52Z |
≈2,355 latched windows in 6.4 days ≈ 1 in every 4 minutes has a 60 s cross-user bucket with ≥5 writes and ≥50% failures (floor: `MIN_WRITES = 5`, `MIN_FAILURE_RATE = 0.5`, `api/analytics-health.js:21-22`). Sampled event extras show `writeCount` 5–8, `failureRate` 0.5–0.6 — the floor is being crossed at the minimum sample size, not during big bursts.
Client-side captures in the same window (`Umami collector write failed`, per-`failureKind` fingerprints):
| Sentry issue | failureKind | events | users | window |
|---|---|---|---|---|
| WORLDMONITOR-YC (7651107112) | network | 43 | 23 | 08-04 → 08-07 |
| WORLDMONITOR-YD (7651157469) | queue-overflow | 30 | 30 | 08-04 → 08-07 (evidence posted on #6288) |
| WORLDMONITOR-YF (7651226146) | timeout | 16 | 14 | 08-04 → 08-08 |
| WORLDMONITOR-YE (7651197639) | missing-receipt | 10 | 10 | 08-04 → 08-07 |
| WORLDMONITOR-YK (7652554919) | http | 63 | 61 | 08-04 20:28 → 21:41 only (burst, self-resolved) |
## The fork to resolve
1. **Collector still degraded**: mixed kinds (timeout + missing-receipt + http burst) alongside the aggregate floor suggests real write-path failures continuing after the August fixes. Cross-check Railway collector logs / #6024-style memory pressure for 08-01 → 08-08, especially the 08-04 20:28–21:41Z http burst.
2. **Floor mis-sized for low-traffic minutes**: `MIN_WRITES = 5` in a 60 s cross-user bucket is small at off-peak; a handful of ad-blocker/privacy-layer users can own an entire bucket. If (1) comes back clean, the floor needs a higher minimum sample or a sustained-windows requirement before latching.
## Acceptance criteria
- [ ] Attribute the 08-01 → 08-08 alarm volume: real write failures vs environmental noise, with collector-side evidence (not client-side counters alone).
- [ ] Either fix the collector failure mode or resize the floor (`MIN_WRITES` / `MIN_FAILURE_RATE` / sustained-window requirement) so the alarm only fires on actionable degradation.
- [ ] After the change, the alarm rate drops to a level where a single latched window is worth a human look; verify over ≥48 h of production.
- [ ] Existing Sentry issues Y7/Y6 stop accumulating at ~340/day (they are left unresolved as live monitors until then).
## Related
- #6288 (open) — client transport queue-parking; owns the `queue-overflow` slice.
- #5974 (merged 08-02) — previous floor-tuning round; this issue is the evidence that tuning did not converge.
- #6024 / #6053 / #6057 — August collector OOM/pool saga.
Contributor guide
Research direction
Start with api/analytics-health.js, especially MIN_WRITES, MIN_FAILURE_RATE, and the Redis SET NX latch at lines 82-88. Compare the 08-01 to 08-08 Sentry windows with Railway collector logs, including the 08-04 HTTP burst and memory-pressure history. Done means the failure source is attributed, the collector or alarm floor is corrected, and production monitoring shows actionable rates for at least 48 hours.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- redis, typescript
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100