koala73 / koala73/worldmonitor
fix(digest): relay-auth failure exits 0 — a dead digest cron reports green
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Summary
When the digest cron cannot authenticate to the Convex relay, it exits **0**. Railway records the run as a success, so a total delivery outage looks like a healthy green cron.
Observed during the #8208 outage: the 02:38Z run on 2026-09-16 fetched zero rules, sent zero digests, and completed "successfully".
```
2026-09-16T02:38:59Z [inf] [digest] Cron run start: 2026-09-16T02:38:58.902Z
2026-09-16T02:39:03Z [inf] [digest] watchlist scan: hashes=2293 candidates=76 events=0 enqueued=0
2026-09-16T02:39:04Z [err] [digest] Failed to fetch rules: 401
```
That is the entire run. Nothing else fired.
## Cause
`scripts/seed-digest-notifications.mjs:2183-2191` — the non-ok branch logs, writes run meta, and `return`s from `main()`. `main()` resolves, the `.catch` at the bottom of the file never runs, and the process exits 0:
```js
if (!res.ok) {
console.error('[digest] Failed to fetch rules:', res.status);
await writeDigestLastRunMeta({
startedAtMs: nowMs,
status: 'error',
errorReason: `fetch_rules_http_${res.status}`,
});
return;
}
```
The `catch` arm below it (`fetch_rules_failed:*`, `:2192-2200`) has the same shape, so a network failure to Convex is equally silent.
## Impact
The only thing that surfaced the outage was the `digestNotifications` staleness check in `/api/health`, which needs `maxStaleMin 90` to elapse first. Compare `notification-relay`, which logged a 401 per event and stayed loud — that is the behaviour we want here.
This also mattered for triage: the `diagnose-railway-seeders` skill classifies on crash signatures, and a green run emits none.
## Proposed fix
Treat "could not reach or authenticate to the relay" as a failed run: `process.exit(1)` after `writeDigestLastRunMeta`, matching the existing brief-compose gate at the end of `main()` (which already flushes telemetry before exiting). Both the `!res.ok` and the `catch` arm should do it.
Deliberately *not* in scope: the `No digest rules found` path (`:2202`) is a legitimate zero-work run and must stay green.
## Acceptance
- A run whose rules fetch 401s / 5xxs / times out exits non-zero and shows red in Railway.
- A run with zero due rules still exits 0.
- Test covering both, since the difference is one `return` vs `process.exit(1)` and nothing else catches it.
Context: #8208, and the validation comment https://github.com/koala73/worldmonitor/issues/8208#issuecomment-5691556926
Contributor guide
Research direction
Start in scripts/seed-digest-notifications.mjs at the rules-fetch branches around lines 2183-2200, then inspect the existing brief-compose exit path for the intended telemetry flush behavior. Add coverage for relay authentication or network failures and for zero due rules, verifying that failures exit non-zero while zero-work runs exit 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 85/100