koala73 / koala73/worldmonitor
Root-cause the intermittent internal-MCP 401 once failure-mode telemetry lands (WORLDMONITOR-XZ)
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Symptom
A signed internal-MCP tool call intermittently fails with `401 invalid_internal_mcp_signature`. Sentry surfaces it as **WORLDMONITOR-XZ** (`list-global-tenders HTTP 401`), `auth_kind=pro`, `tool=get_procurement_opportunities` — paying customers, on a paid tool.
## The real blast radius is wider than Sentry shows
Axiom `wm_api_usage`, filtered to `user_agent startswith 'worldmonitor-mcp-edge'`, 30 days to 2026-09-13:
| route | 401 / anon / auth_401 |
|---|---|
| `/api/economic/v1/list-global-tenders` | 3 |
| `/api/intelligence/v1/get-country-risk` | 3 |
| `/api/infrastructure/v1/get-bootstrap-data` | 1 |
Only `list-global-tenders` reaches Sentry, because its tool calls `assertToolFetchOk` and raises. The other two **swallow the 401 and return empty** — the caller sees no data rather than an error. So the Sentry issue understates this, and there may be more routes affected that never surface at all.
For scale: the same proxy succeeds in the hundreds over the same window (748 `get-country-risk`, 512 `get-intel-timeline`). This is rare and transient, not a broken path.
## Why it was not fixed directly
`server/gateway.ts` deliberately collapses **five** distinct failures into one indistinguishable 401:
- timestamp outside the ±30s `INTERNAL_MCP_TIMESTAMP_WINDOW_SECONDS`
- malformed signature envelope
- missing `X-WM-MCP-User-Id`
- failed HMAC compare
- replay-nonce already spent
That is correct security design — a caller who can tell those apart has a forgery oracle — but it also made the failure **unreproducible**. `auth_kind=anon` on the gateway side only says "no identity resolved". Any fix would have been a guess.
## What has been done
PR #8086 ships the diagnostic: a distinct server-side `reason` per failure mode (`internal_mcp_ts_window`, `internal_mcp_sig_mismatch`, `internal_mcp_replay`, `internal_mcp_no_user`, `internal_mcp_malformed_sig`, `internal_mcp_bad_request`), with the caller-facing response held byte-identical and pinned by a test. No behavior changed.
## What this issue tracks
**After #8086 has been live ~1 week**, run:
```
wm_api_usage
| where reason startswith 'internal_mcp_'
| summarize count() by reason, route, bin(_time, 1d)
```
The distribution names the cause, and the real fix follows:
- **`internal_mcp_ts_window` dominant** → clock skew between the MCP edge and the gateway. Widening the window is the obvious lever but weakens replay protection; the replay-cache TTL (`2 * WINDOW + 5`) must move with it.
- **`internal_mcp_replay` dominant** → something is re-sending an already-signed request. A transport-level retry of an idempotent GET would do this; an application-level retry would not (it re-signs with a fresh nonce). Compare with the checkout-retry/idempotency-lock race, which is the same shape.
- **`internal_mcp_sig_mismatch` dominant** → genuine signer/verifier divergence. Suspect a deploy window where the MCP edge function and the gateway run different deployments with different `MCP_INTERNAL_HMAC_SECRET` values; correlate the timestamps against deploy times.
Worth also deciding separately whether `get-country-risk` and `get-bootstrap-data` **should** be swallowing this — a 401 that silently becomes an empty result is its own bug, independent of the root cause.
Refs WORLDMONITOR-XZ. Follow-up to #8086.
https://claude.ai/code/session_012nVNfJrGEQwfhB3keoy73H
Contributor guide
Research direction
Start with server/gateway.ts and PR #8086, then wait until its diagnostic has been live for about a week and run the provided wm_api_usage query. Correlate the dominant reason with deployments or retry behavior, then determine the appropriate fix and whether the other routes should stop swallowing 401 responses. Done means the root cause and affected routes are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, observability, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100