cloudflare / cloudflare/workers-sdk
wrangler dev: steady ~5s request cadence phase-locks with kj's 5s keep-alive timeouts — POSTs intermittently fail with 503 "worker restarted mid-request" (worker never restarted)
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.5k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 187
Description
### Summary
`wrangler dev` intermittently fails POST requests with `503 "Your worker restarted mid-request"` even though the worker never restarts. The trigger is surprisingly specific: **a client sending requests on a steady ~5.000s cadence phase-locks with KJ's two default 5-second connection timeouts inside the local dev proxy chain**, so every request rides exactly on the millisecond where an idle keep-alive connection is being closed by the other side.
We hit this in production-like local dev (a timer service firing every 5s) with **30–47% of requests failing**. Changing the cadence to 4s or 6s drops the failure rate to **0%**.
This is the *request-drop mechanism* underlying the misleading 503. It is related to but distinct from PR #14593, which fixes the error *classification* (origin vs href comparison) — with that PR merged, the error becomes honest, but the request still fails.
### Minimal reproduction
Any worker project (the failure happens in the ProxyWorker forwarding layer, worker code is irrelevant):
```bash
npx wrangler dev --port 8787 # any worker
node repro.mjs 8787 5000 # steady 5s cadence -> ~40% failures
node repro.mjs 8787 4000 # 4s -> 0%
node repro.mjs 8787 6000 # 6s -> 0%
```
```js
// repro.mjs — POST wrangler dev on a steady cadence (send-time aligned)
import http from "node:http";
const port = Number(process.argv[2] ?? 8787);
const interval = Number(process.argv[3] ?? 5000);
let sent = 0, failed = 0;
function shoot() {
const body = Buffer.from(JSON.stringify({ hello: "world" }));
const req = http.request(
{ hostname: "127.0.0.1", port, path: "/", method: "POST",
headers: { "content-type": "application/json", "content-length": body.length } },
(res) => {
res.resume();
res.on("end", () => {
sent++;
if (res.statusCode >= 500) failed++;
console.log(new Date().toISOString(), res.statusCode, `sent=${sent} failed=${failed} reused=${req.reusedSocket}`);
});
});
req.on("error", (e) => { sent++; failed++; console.log(new Date().toISOString(), "SOCKET-ERR", e.code, `sent=${sent} failed=${failed}`); });
req.end(body);
}
shoot();
setInterval(shoot, interval);
```
Measured on our project (wrangler 4.99.0; an earlier investigation also reproduced on 4.108.0):
| Client | Cadence | Result |
| --- | --- | --- |
| node, keep-alive globalAgent | 5000ms (send-time aligned) | **43.3% failed** (30 shots: 9× HTTP 503 + 4× ECONNRESET) |
| node, new connection per request | 5000ms | **40.0% failed** (20 shots: 8× 503, 0× ECONNRESET) — inbound reuse is *not* required |
| node, keep-alive | **4000ms** | 0% (20 shots) |
| node, keep-alive | **6000ms** | 0% (20 shots) |
| bun fetch | 5000ms | 33.3% failed |
| any client | 5s measured *after* the response (i.e. interval ≈ 5015ms) | 0% — the ~15ms systematic offset de-phases the race |
### Mechanism (directly observed at TCP level)
We placed a transparent TCP observer proxy between ProxyWorker and the UserWorker socket (by temporarily patching the inner-fetch port in `ProxyWorker.js`). Failing request, millisecond timestamps:
```text
48706 C3 OPEN (ProxyWorker opens inner connection)
48706 C3 DATA A->B 960B <- previous request
48712 C3 DATA B->A 72B <- previous response, fine
53706 C3 DATA A->B 960B <- THIS request (connection idle for exactly 5000ms, pool reuses it)
53706 C3 CLOSE by-B <- same millisecond: UserWorker closes the idle connection. No response.
```
Why exactly 5 seconds, twice:
- `kj/compat/http.h`: `HttpServerSettings.pipelineTimeout = 5 * kj::SECONDS` (server closes an idle keep-alive connection 5s after the last request) and `HttpClientSettings.idleTimeout = 5 * kj::SECONDS` (client pool keeps an idle connection up to 5s). workerd does not override either default.
- The UserWorker closes the inner connection at `last request + 5.000s`; a client on a 5.000s cadence makes ProxyWorker's next reuse attempt land on that exact millisecond. The ±1ms jitter between the two clocks decides each shot — hence the ~"coin-flip" failure rates and the bursty all-green/all-red streaks.
- The failed inner `fetch()` rejects with kj `DISCONNECTED`, which `decodeTunneledException` (workerd `src/workerd/jsg/util.c++`) renders as `"Network connection lost."`. ProxyWorker's catch then misclassifies it (see #14593) and synthesizes the 503 with the misleading "worker restarted" text. GET/HEAD are silently retried, so only non-GET traffic surfaces it.
- The same race on the *inbound* connection (client → ProxyWorker, same 5s pipelineTimeout) produces the ECONNRESET flavor: the client reuses a socket the proxy just closed. Any keep-alive reverse proxy in front of wrangler dev turns that into a 502.
A steady ~5s cadence is not exotic — it is a default polling/heartbeat interval in lots of tooling, and anyone hitting this sees only the misleading "restarted mid-request" text (we lost two days to it before instrumenting).
### Suggested fix directions
1. **Retry once on stale-connection reuse** in ProxyWorker's inner fetch (or in kj's pooled HttpClient): connection was reused + zero response bytes received → replay on a fresh connection. This is what curl, Go's net/http, undici and browsers do for this classic race; it would eliminate both the 503 and the drop.
2. Alternatively (cheaper): disable keep-alive / set `Connection: close` on the ProxyWorker→UserWorker hop in local dev — the loopback reconnect cost is negligible compared to a 40% failure rate at unlucky cadences.
3. Merge #14593 regardless, so that whatever still fails is at least reported truthfully instead of as a phantom worker restart.
### Environment
- wrangler 4.99.0 (earlier investigation also reproduced on 4.108.0), local dev (`wrangler dev`, miniflare/workerd)
- macOS (darwin 25.4.0), node v24.13.1, bun 1.3.9
- Reproduces with and without a reverse proxy in front; with keep-alive and non-keep-alive inbound clients; worker code irrelevant
Contributor guide
Research direction
Run the supplied repro.mjs against wrangler dev at 5000ms, then compare 4000ms and 6000ms cadences. Read ProxyWorker.js, kj/compat/http.h, and workerd src/workerd/jsg/util.c++ to trace stale connection reuse and the resulting error classification. Done means the 5-second POST repro no longer drops requests or reports a misleading worker-restarted error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, node.js, typescript
- Domain
- backend, cli, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100