koala73 / koala73/worldmonitor

ops(analytics): alert on umami memory, not Railway deployment status — status has masked two silent collector deaths

Open
#6,025 2 comments 0 reactions 0 assignees View on GitHub
area: infrastructure enhancement P1
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Problem

The Railway `umami` service has now died silently **twice**, and in both incidents Railway's own status was useless:

- **2026-07-20 → 07-24** (#5565): 4 days dark, ~285k events/day lost. Railway showed `SUCCESS` throughout.
- **2026-08-01**: two OOM crashes, 144 min dark, ~150k events lost. Railway showed the **same** `SUCCESS` deployment (created `2026-07-24T20:36:36`) across both.

The container restarts in place, so no deployment row is written and the status never flips. **Deployment status can never detect this failure mode.**

## What actually works

The `MEMORY_USAGE_GB` metric is unambiguous — it traced the 2026-08-01 crashes to the minute:

```
18:00–18:40 flat 0.40 GB healthy
19:56 5.3 GB writes stop
20:48 9.12 GB ceiling
20:52 0.45 GB crash + restart
23:44–00:00 9.20–9.21 GB pinned, GC thrash
00:04 0.27 GB crash + restart
```

Healthy baseline sits at **0.34–0.67 GB**; the ceiling is `--max-old-space-size=8192` (~9.2 GB RSS). That is a wide, unambiguous band to alarm on.

## Proposal

Alert on the umami service's memory metric, not its deployment status. Available via the Railway GraphQL API:

```graphql
query M($start: DateTime!, $end: DateTime!, $envId: String!, $svcId: String!,
$m: [MetricMeasurement!]!, $w: Int) {
metrics(startDate: $start, endDate: $end, environmentId: $envId,
serviceId: $svcId, measurements: $m, averagingWindowSeconds: $w) {
measurement values { ts value }
}
}
```

- service id `d7620480-e05a-4c09-b210-05166c3c0e59`, env `91a05726-0b83-4d44-a33e-6aec94e58780`
- measurement `MEMORY_USAGE_GB`
- note: `step` is **not** a valid argument (use `averagingWindowSeconds`), and schema introspection is 403-blocked

A sustained read above ~4 GB means the leak has engaged and a crash is coming — that is roughly **50 minutes of lead time** based on the 08-01 climb rate, enough to act before any data is lost.

## Complementary cheap check

Write-lag against the Umami Postgres is the authoritative liveness signal and costs one query:

```sql
select extract(epoch from (now() - max(created_at))) from website_event;
```

Any value beyond a few minutes during active traffic means the collector is not accepting writes.

Do **not** rely on `abacus.worldmonitor.app/api/heartbeat` alone: it returns **403 to a bare curl UA** (Cloudflare WAF generic-UA rule) and 200 to a browser UA, so a naive monitor alarms on a healthy collector. Distinguish by latency — fast 403 = WAF, 502 after ~15s = origin dead.

Root cause of the leak itself is tracked separately.

Contributor guide

Open the contributing guide

Research direction

No implementation file or test is named. Start by locating the existing Railway deployment-status monitor and its alerting entry point, then review the Railway GraphQL metrics query and the Umami Postgres write-lag query. Done means sustained MEMORY_USAGE_GB and write-lag alerts detect collector failure without relying on deployment status or a bare heartbeat request.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, typescript
Domain
databases, devops, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.