koala73 / koala73/worldmonitor

ops(edge): Cloudflare 520-527 cluster at minute :30 of every hour, upstream of the Vercel origin

Open
#6,410 0 comments 0 reactions 0 assignees View on GitHub
bug P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Split out of #6370 so the signal is not lost when that issue closes. #6409 fixes the Sentry mis-grouping there; this is the remaining, unexplained part.

## The finding

Cloudflare `520-527` errors on worldmonitor.app fire disproportionately at **minute :30-:31 of the hour**, on every hour, across multiple routes.

Complete population of `kind:api_cf_5xx` events over 90d (the tag has existed since #3854 landed 2026-05-22, so this is the full history):

| scope | at :30-:31 | total | share |
|---|---|---|---|
| `/api/economic/v1/get-fred-series-batch` | 14 | 25 | 56% |
| all routes | 16 | 41 | 39% |

Uniform expectation is 2/60 = 3.3%.

## Why it is not an artifact

Each of these was checked:

- **Arrivals are uniform.** Axiom `wm_api_usage`, 7d, this route: request counts per minute-of-hour range 527-778 with no peak at :30 (minute 30 = 575, minute 31 = 631; the max is minute 45 at 778). So the failures cluster, the traffic does not.
- **Origin latency and volume at :30 are unremarkable.** 14d on this route, status 200: p50 192ms, p95 ~470-560ms, p99 ~656ms, zero cold starts, flat across all 60 buckets.
- **The origin never sees them.** 7d on this route Axiom records 36,938 requests with statuses 200/401/429/403/405 and **zero 5xx**. Cloudflare generates these without the request reaching Vercel.
- **Not a client refresh cadence.** `REFRESH_INTERVALS.fred` is `6 * 60 * 60 * 1000`, a `setInterval` relative to page load, not wall-clock aligned.
- **Not one of our crons.** The Railway `*/15` and `*/5` seeders and the `*/30` `deploy-gate.yml` all fire at :00 too, and :00 is clean (1 event in 90d). `deploy-gate.yml` also only calls the GitHub API, never production.
- **Not one bad client.** Affected users span Chrome / Chrome Mobile / Edge on Linux, Android and Windows, on both `www` and `finance`.

## Secondary signal: geography

The same 41 events are heavily concentrated in one corridor: IN 23, PK 3, AE 3, BD 2, plus TW / VN / ID — 80% Asia/Middle East. That points at specific Cloudflare PoPs (BOM / DEL / DXB) and their path to the Vercel origin region.

## Scale

Low. 25 events on the busiest route over 79 days against ~417k requests is roughly **0.006%**, about 1 in 17,000. Each one costs a user one panel falling back to cached data via the circuit breaker. This is a "understand it before it grows" ticket, not a live incident.

## Next step

The evidence needed is Cloudflare-side and is the one source not reachable from this repo's telemetry:

1. Cloudflare **Logpush** or the GraphQL Analytics API, `httpRequestsAdaptiveGroups`, filtered to `edgeResponseStatus in (520..527)`, grouped by minute-of-hour and by `coloCode`. Confirm the :30 concentration independently and identify whether it is PoP-specific.
2. If it localizes to BOM/DEL/DXB, check Cloudflare's origin-connection settings for the zone and whether a keep-alive/connection-reuse race with Vercel's edge explains it.
3. Worth ruling out: any hourly automation against the zone (a config push, a cache purge, a DNS/tiered-cache change) that lands on the half hour.

## If it turns out to be unfixable upstream

The fallback is a single retry on `520-527` in `reportServerError`'s caller. It is safe by construction — these never reach the origin, so no side effect can have occurred, and the affected RPCs are read-only POSTs. It was deliberately **not** included in #6409 because a request-path behavior change was not justified at 1-in-17,000 with the circuit breaker already covering the user impact.

## Acceptance criteria

- The `:30` concentration is either explained, or confirmed absent from Cloudflare-side data (which would make it a Sentry-side timestamp artifact).
- A decision recorded on whether to add the 520-527 retry.

Found while diagnosing #6370.

Contributor guide

Open the contributing guide

Research direction

Start by checking the Cloudflare Logpush or GraphQL Analytics data described in the issue, grouped by minute-of-hour and coloCode. Then inspect the reportServerError caller and deploy-gate.yml context before deciding whether the 520-527 retry is warranted. Done means the :30 pattern is explained or disproven and the retry decision is recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.