cloudflare / cloudflare/workers-sdk
containers: wrangler dev never issues the workload container create on native Linux — snapshot-volume step stalls with no diagnostic, surfaces only as kj/timer.c++:30 operation timed out
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.5k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 186
Description
### What versions & operating system are you using?
- wrangler: `4.127.0` (also reproduced on `4.115.0` and `4.116.0`)
- miniflare: `5.20260826.0-alpha`
- Docker CE `29.5.1`, API `1.54` — **native daemon, not Docker Desktop**; reproduced under both **rootful** and **rootless** modes
- Arch Linux, kernel `7.0.9`, x86_64
- `compatibility_date` `2026-07-01`, `compatibility_flags` `["nodejs_compat"]`
### Describe the Bug
On native Linux, `wrangler dev` never asks Docker to create the workload container. The image builds, the `proxy-everything` sidecar is created and started successfully, and then the runtime stops making requests. ~60s later:
```
✘ [ERROR] e = kj/timer.c++:30: overloaded: operation timed out
✘ [ERROR] Uncaught Error: internal error; reference = ...
```
Through `@cloudflare/containers` the same failure reads `Failed to start container: Container failed to start`.
The error mentions neither containers, nor Docker, nor privileges, nor the snapshot-volume step it actually died in — which makes this expensive to diagnose. That diagnostic gap is arguably the more important half of this report.
### Reproduction
Reproduces with **`cloudflare/templates/containers-template`, unmodified**:
```bash
curl -fsSL https://codeload.github.com/cloudflare/templates/tar.gz/refs/heads/main \
| tar xz --strip-components=2 templates-main/containers-template
cd containers-template && npm install && npx wrangler dev
curl localhost:8787/container/testing # => Failed to start container: Container failed to start
```
### Where it stalls
I interposed `socat` on the Docker socket and captured every request/response. **Docker refuses nothing.** ~20 calls, all 2xx, except three benign `404`s (existence checks for a container that legitimately does not exist yet). Abbreviated:
```
GET /v1.54/images/cloudflare-dev/:/json => 200 image IS present
POST /v1.54/images/create?fromImage=...proxy-everything... => 200
GET /v1.54/info => 200
GET /volumes?filters={"name":["workerd-snap-"]} => 200 empty list
GET /containers//json => 404 (expected)
POST /containers/create?name=-proxy => 201 sidecar
POST /containers/-proxy/start => 204 sidecar starts fine
DELETE /containers/?force=true => 404 (expected)
--- nothing further ---
```
After querying for `workerd-snap-` volumes it never issues `POST /volumes/create`, never creates the temp/clone-helper containers, and never creates the workload container.
Notably, workerd's binary carries a distinct error string for **every** failure along that path — `Failed to create Docker volume '...'`, `Failed to create temp container for volume '...'`, `Failed to create snapshot clone helper container for volume '...'`, `Failed to start snapshot clone helper container '...'`, `Failed to inspect Docker image '...'` — and **none of them fire**. So the stall appears to be upstream of workerd's own error handling. `MINIFLARE_WORKERD_V=1` and `--log-level debug` add nothing.
workerd *can* create containers here — it creates and starts the sidecar via its own `POST /containers/create` in the same session.
### What I ruled out (so you don't have to)
- **Not the config.** Cloudflare's own template fails identically.
- **Not Docker.** Every API call succeeds; Docker is never given the chance to refuse.
- **Not the Docker version.** 29.5.1 builds every image and runs the rootless daemon correctly.
- **Not FUSE privileges.** I patched `containerPrivilegesAllowed()` to log its decision. Under rootless it already returns **true**:
```
secOpts=["name=seccomp,profile=builtin","name=rootless","name=cgroupns"]
rootless=true localVm=false rootlessLinux=true DECISION=true
```
Patching it to return `true` unconditionally changes nothing. (Aside: rootful Linux is excluded by that gate by construction, but rootful Docker *does* grant those exact privileges when asked — `docker run --cap-add SYS_ADMIN --device /dev/fuse:/dev/fuse:rwm --security-opt apparmor:unconfined` succeeds. So the gate may be stricter than it needs to be, though that is not the cause of this bug.)
- **Not `cloudflare/containers#155`.** The `cloudflare-dev/*` image is present and returns `200 OK`.
### A secondary finding: `dev.container_engine` silently requires a URI scheme
Setting `"container_engine": "/var/run/docker.sock"` — a bare path, which is what [the configuration docs](https://developers.cloudflare.com/workers/wrangler/configuration/) describe ("setting the socket path") — fails. The value is passed through verbatim to `socketPath` and parsed with `new URL()` by both `isLocalDockerEndpoint` and workerd, so it must carry a `unix://` scheme. Miniflare's own internal default is `unix:///var/run/docker.sock`.
With a bare path, workerd treats it as a hostname:
```
DNS lookup failed.; params.host = /var/run/docker.sock; gai_strerror(status) = Name or service not known
```
and `isLocalDockerEndpoint` returns false, silently refusing container privileges as a side effect. On wrangler 4.115 this produced no message at all. Either the docs should show the `unix://` form, or the setting should accept a bare path.
### Possibly related
- #13672 — same native-Linux-Docker-CE environment; maintainer could not reproduce. **I can independently confirm its premise on this machine:** after the pull, `docker images` shows `cloudflare/proxy-everything:` — the tag is dropped exactly as described. In my case the sidecar still starts, so it is not what breaks this, but the tag-dropping behaviour is real outside CI.
- #155 (cloudflare/containers), #10221 — adjacent local-dev container failures, both macOS.
Happy to run further diagnostics, apply a patched build, or test a pre-release — I have the reproduction and the socket-level tracing set up.
Contributor guide
Research direction
Reproduce the failure with the unmodified cloudflare/templates/containers-template and trace Docker socket requests after the workerd-snap volume query. Inspect containerPrivilegesAllowed, isLocalDockerEndpoint, and the dev.container_engine parsing path, then compare the behavior with the documented socket setting. Done means native Linux reaches workload container creation or reports the failed snapshot-volume step with a specific diagnostic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- cli, devops
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100