koala73 / koala73/worldmonitor
ops(analytics): alert on umami memory, not Railway deployment status — status has masked two silent collector deaths
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Problem
The Railway `umami` service has now died silently **twice**, and in both incidents Railway's own status was useless:
- **2026-07-20 → 07-24** (#5565): 4 days dark, ~285k events/day lost. Railway showed `SUCCESS` throughout.
- **2026-08-01**: two OOM crashes, 144 min dark, ~150k events lost. Railway showed the **same** `SUCCESS` deployment (created `2026-07-24T20:36:36`) across both.
The container restarts in place, so no deployment row is written and the status never flips. **Deployment status can never detect this failure mode.**
## What actually works
The `MEMORY_USAGE_GB` metric is unambiguous — it traced the 2026-08-01 crashes to the minute:
```
18:00–18:40 flat 0.40 GB healthy
19:56 5.3 GB writes stop
20:48 9.12 GB ceiling
20:52 0.45 GB crash + restart
23:44–00:00 9.20–9.21 GB pinned, GC thrash
00:04 0.27 GB crash + restart
```
Healthy baseline sits at **0.34–0.67 GB**; the ceiling is `--max-old-space-size=8192` (~9.2 GB RSS). That is a wide, unambiguous band to alarm on.
## Proposal
Alert on the umami service's memory metric, not its deployment status. Available via the Railway GraphQL API:
```graphql
query M($start: DateTime!, $end: DateTime!, $envId: String!, $svcId: String!,
$m: [MetricMeasurement!]!, $w: Int) {
metrics(startDate: $start, endDate: $end, environmentId: $envId,
serviceId: $svcId, measurements: $m, averagingWindowSeconds: $w) {
measurement values { ts value }
}
}
```
- service id `d7620480-e05a-4c09-b210-05166c3c0e59`, env `91a05726-0b83-4d44-a33e-6aec94e58780`
- measurement `MEMORY_USAGE_GB`
- note: `step` is **not** a valid argument (use `averagingWindowSeconds`), and schema introspection is 403-blocked
A sustained read above ~4 GB means the leak has engaged and a crash is coming — that is roughly **50 minutes of lead time** based on the 08-01 climb rate, enough to act before any data is lost.
## Complementary cheap check
Write-lag against the Umami Postgres is the authoritative liveness signal and costs one query:
```sql
select extract(epoch from (now() - max(created_at))) from website_event;
```
Any value beyond a few minutes during active traffic means the collector is not accepting writes.
Do **not** rely on `abacus.worldmonitor.app/api/heartbeat` alone: it returns **403 to a bare curl UA** (Cloudflare WAF generic-UA rule) and 200 to a browser UA, so a naive monitor alarms on a healthy collector. Distinguish by latency — fast 403 = WAF, 502 after ~15s = origin dead.
Root cause of the leak itself is tracked separately.
Contributor guide
Research direction
No implementation file or test is named. Start by locating the existing Railway deployment-status monitor and its alerting entry point, then review the Railway GraphQL metrics query and the Umami Postgres write-lag query. Done means sustained MEMORY_USAGE_GB and write-lag alerts detect collector failure without relying on deployment status or a bare heartbeat request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, typescript
- Domain
- databases, devops, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100