cloudflare / cloudflare/workers-sdk

wrangler dev with remote bindings: persistent "internal error; reference =" from a long-lived session, escalating after ~2h to ProxyWorker "Network connection lost." and exit code 1

Open
#15,421 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
4.5k
Forks
1.5k
Avg merge
3d 8h
Merged PRs (30d)
186

Description

## Versions and environment

- wrangler: 4.116.0 (miniflare 4.20260730.0)
- node: v22.22.0, pnpm, Ubuntu Linux (systemd service, non-interactive)
- Worker: `wrangler dev --config wrangler.dev.jsonc --port ... --inspector-port ... --persist-to ...`
- The config has **20 bindings with `"remote": true`**: one D1 database, several KV namespaces, seven R2 buckets, and a Workers for Platforms dispatch namespace. The rest of the config (service bindings, queues, AE datasets) is local.

Before filing this I diffed `packages/remote-bindings` (src and templates) between the `wrangler@4.116.0` and `wrangler@4.127.1` tags: apart from #15142 (unique entrypoint suffix for binding-only updates) and the miniflare v5 config-shape adaptation from #14994, the package is byte-identical, so I believe this reproduces on current wrangler too.

## Describe the bug

A long-lived `wrangler dev` session with remote bindings emits a steady stream of

```
✘ [ERROR] Error: internal error; reference =
```

starting about 90 seconds after boot, at a roughly constant 25 to 30 lines per minute, often in clusters of three (this part looks like #15351). User-facing requests keep returning 200 for a long time, but remote-binding-backed requests progressively degrade, and after about two hours the session dies entirely: the ProxyWorker reports `Network connection lost.` and wrangler exits with code 1, printing a final `✘ [ERROR]` with an empty message.

We run four such dev sessions as systemd services and have hit this twice in the first day. In one earlier incident the session did not exit but instead reached a state where **every** remote-binding call failed with `internal error; reference =` until the process was restarted. A restart always recovers cleanly.

## Timeline from one session (wrangler debug log `wrangler-2026-08-29_14-16-02_949.log`)

- `14:16:05Z` wrangler 4.116.0 starts (`⎔ Establishing remote connection...` at boot, once).
- `14:17:35Z` first error: `internal error; reference = qkl6mh6cfpokrfogc6dbgte8`, immediately followed by `tosdko4jmuu6uur3i2nctd37` and `vgr2ci2eo3u5d0ft9sff61qr`.
- `14:17 to 16:32` a further 3,675 `internal error; reference =` lines at a near constant rate. D1-backed endpoints that answer in low hundreds of ms when healthy degrade to 1.6 to 4.6 s, and callers of this Worker queue up to 97 s.
- `16:32:10.507Z` (from the debug log):

```
Error in ProxyController: Error inside ProxyWorker
Error
at castErrorCause (.../wrangler/wrangler-dist/cli.js:166823:19)
at ProxyController2.emitErrorEvent (.../cli.js:260798:20)
at ProxyController2.onProxyWorkerMessage (.../cli.js:260675)
at PROXY_CONTROLLER (.../cli.js:260403:24)
at async #handleLoopbackCustomFetchService (.../miniflare/dist/src/index.js:103327:22)
at async #handleLoopback (.../miniflare/dist/src/index.js:103561:20) {
cause: {
name: 'Error',
message: 'Network connection lost.',
stack: 'Error: Network connection lost.'
}
}
```

- `16:32:10.610Z` final output:

```
✘ [ERROR]

Note that there is a newer version of Wrangler available (4.127.1). Consider checking whether upgrading resolves this error.
```

and the process exits with code 1 (`provisioning exited (code=1, signal=null)` from our supervisor).

Last reference id before death: `7pnpa373iv6l8duqgt80ia5q`. A few more from the final minutes, in case they are useful to hand to the workerd or D1 teams: `a3ijvbbo5b3fbhtvgvm8ar0e`, `mlhsjjdqpsq81gr0rknnb0d1`, `r1hcj65oe5kalj95ed4rko0e`, `is0i85f302ikr0aor7uutkr8`, `q5uajrpbqq7bp8hpmf8m8mvq`, `9ga5p6a9redmj9ghkfqi911i`. I can paste a fresh set or share the full 30k-line debug log on request.

## Expected behavior

Remote bindings should survive a long-lived dev session, or re-establish the session when it goes bad. An error storm on the remote-binding path should not end in the whole `wrangler dev` process exiting.

## Two resilience gaps I noticed while reading `packages/remote-bindings` at 4.127.1

I cannot prove either of these is the cause of the storm above (the errors here start 90 seconds in, well before any token expiry), but both look like they would turn a transient failure into the permanent "every remote-binding call fails until restart" state we saw in the other incident:

1. **A single failed proactive token refresh permanently disarms refreshing.** `RemoteRuntimeController.#scheduleRefresh` is only re-armed from inside `#updatePreviewToken` on success. If the refresh at the 50-minute mark (`PREVIEW_TOKEN_REFRESH_INTERVAL`) fails once, `#refreshPreviewToken` emits an error event and never reschedules, so the preview token expires at the 1-hour mark and stays expired. The error event itself is only visible at debug level: `DevEnv`'s constructor installs a `logger.debug` "error" listener, and `startRemoteProxySession` removes its own error listener once startup completes, so from the user's point of view the session silently rots.

2. **Reactive expiry detection only matches one response shape.** `checkForPreviewTokenError` in `templates/startDevWorker/ProxyWorker.ts` triggers `previewTokenExpired` only on status 400 with a body containing `Invalid Workers Preview configuration` or `error code: 1031`. A remote-binding call that fails with a 500-class `internal error; reference = ...` never triggers re-establishment, so once a session is bad for any other reason it never heals.

## Related

- #15351 (same `internal error; reference =` clusters of three against remote D1 while requests still return 200; this report adds the long-session degradation and eventual process death)

Contributor guide

Open the contributing guide

Research direction

Start in packages/remote-bindings and templates/startDevWorker/ProxyWorker.ts; trace RemoteRuntimeController.#scheduleRefresh, #updatePreviewToken, checkForPreviewTokenError, and startRemoteProxySession against the debug timeline. Reproduce a long-lived session with the listed remote bindings and inspect the referenced log patterns. Done means transient remote-binding failures recover or are surfaced without ending wrangler dev, with regression coverage for the observed failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
cli, devtools, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.