koala73 / koala73/worldmonitor
ops(edge): Cloudflare 520-527 cluster at minute :30 of every hour, upstream of the Vercel origin
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Split out of #6370 so the signal is not lost when that issue closes. #6409 fixes the Sentry mis-grouping there; this is the remaining, unexplained part.
## The finding
Cloudflare `520-527` errors on worldmonitor.app fire disproportionately at **minute :30-:31 of the hour**, on every hour, across multiple routes.
Complete population of `kind:api_cf_5xx` events over 90d (the tag has existed since #3854 landed 2026-05-22, so this is the full history):
| scope | at :30-:31 | total | share |
|---|---|---|---|
| `/api/economic/v1/get-fred-series-batch` | 14 | 25 | 56% |
| all routes | 16 | 41 | 39% |
Uniform expectation is 2/60 = 3.3%.
## Why it is not an artifact
Each of these was checked:
- **Arrivals are uniform.** Axiom `wm_api_usage`, 7d, this route: request counts per minute-of-hour range 527-778 with no peak at :30 (minute 30 = 575, minute 31 = 631; the max is minute 45 at 778). So the failures cluster, the traffic does not.
- **Origin latency and volume at :30 are unremarkable.** 14d on this route, status 200: p50 192ms, p95 ~470-560ms, p99 ~656ms, zero cold starts, flat across all 60 buckets.
- **The origin never sees them.** 7d on this route Axiom records 36,938 requests with statuses 200/401/429/403/405 and **zero 5xx**. Cloudflare generates these without the request reaching Vercel.
- **Not a client refresh cadence.** `REFRESH_INTERVALS.fred` is `6 * 60 * 60 * 1000`, a `setInterval` relative to page load, not wall-clock aligned.
- **Not one of our crons.** The Railway `*/15` and `*/5` seeders and the `*/30` `deploy-gate.yml` all fire at :00 too, and :00 is clean (1 event in 90d). `deploy-gate.yml` also only calls the GitHub API, never production.
- **Not one bad client.** Affected users span Chrome / Chrome Mobile / Edge on Linux, Android and Windows, on both `www` and `finance`.
## Secondary signal: geography
The same 41 events are heavily concentrated in one corridor: IN 23, PK 3, AE 3, BD 2, plus TW / VN / ID — 80% Asia/Middle East. That points at specific Cloudflare PoPs (BOM / DEL / DXB) and their path to the Vercel origin region.
## Scale
Low. 25 events on the busiest route over 79 days against ~417k requests is roughly **0.006%**, about 1 in 17,000. Each one costs a user one panel falling back to cached data via the circuit breaker. This is a "understand it before it grows" ticket, not a live incident.
## Next step
The evidence needed is Cloudflare-side and is the one source not reachable from this repo's telemetry:
1. Cloudflare **Logpush** or the GraphQL Analytics API, `httpRequestsAdaptiveGroups`, filtered to `edgeResponseStatus in (520..527)`, grouped by minute-of-hour and by `coloCode`. Confirm the :30 concentration independently and identify whether it is PoP-specific.
2. If it localizes to BOM/DEL/DXB, check Cloudflare's origin-connection settings for the zone and whether a keep-alive/connection-reuse race with Vercel's edge explains it.
3. Worth ruling out: any hourly automation against the zone (a config push, a cache purge, a DNS/tiered-cache change) that lands on the half hour.
## If it turns out to be unfixable upstream
The fallback is a single retry on `520-527` in `reportServerError`'s caller. It is safe by construction — these never reach the origin, so no side effect can have occurred, and the affected RPCs are read-only POSTs. It was deliberately **not** included in #6409 because a request-path behavior change was not justified at 1-in-17,000 with the circuit breaker already covering the user impact.
## Acceptance criteria
- The `:30` concentration is either explained, or confirmed absent from Cloudflare-side data (which would make it a Sentry-side timestamp artifact).
- A decision recorded on whether to add the 520-527 retry.
Found while diagnosing #6370.
Contributor guide
Research direction
Start by checking the Cloudflare Logpush or GraphQL Analytics data described in the issue, grouped by minute-of-hour and coloCode. Then inspect the reportServerError caller and deploy-gate.yml context before deciding whether the 520-527 retry is warranted. Done means the :30 pattern is explained or disproven and the retry decision is recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100