bug(envd): rare startup race — MMDS poller keeps running but HTTP listener never binds (sandbox refuses :49983 instantly, orchestrator can't tell)

Summary

On a self-hosted E2B-compatible deployment we occasionally observe sandboxes where the envd process is clearly alive — Firecracker's fc_api log shows a steady Get /mmds → 200 every 5–15 s — but a plain curl from inside the sandbox netns is refused instantly with no listener:

GET http://<vm-ip>:49983/       -> Failed to connect after 0 ms: Couldn't connect to server
GET http://<vm-ip>:49983/health -> Failed to connect after 0 ms: Couldn't connect to server

A healthy sandbox on the same host, same template, same veth topology answers HTTP 404 in ~0.4 ms / HTTP 204 in ~0.4 ms immediately.

The stuck sandbox stays in this state for the rest of its life. Because envd is running under a runsv-like supervisor and never exits, the supervisor does not restart it. Any orchestrator that only waits for the Firecracker API socket (not for envd:49983) marks the sandbox Running, so every subsequent proxy request fails with hyper/Go connect: connection refused.

Rate: sparse — well under 1 % of fresh cold boots. Not tied to any single template — the same template runs healthy on sibling hosts / other sandboxes.

What this is NOT

• Not #1478 / envd-clock-sync fix. That variant is 100 % reproducible and manifests as read-RST after a successful TCP accept (a listener exists). This is connect-refused: the guest kernel returns RST from the SYN because no socket is listening on 49983.
• Not #3256 (Start deadlock). That variant leaves the HTTP mux fully alive (/health → 204 in <1 ms) and only wedges Process.* / Filesystem.*. Here /health itself is refused instantly.
• Not #3559 (MMDS-poll goroutine growth). Fc_api MMDS traffic looks normal — one Get every 5–15 s, not a storm.
• Not networking. The MMDS poller path from the guest works (poller replies land), a healthy sibling sandbox on the same host with the same tap/veth layout responds instantly, and ip route inside the netns is identical between healthy and stuck. The guest TCP stack is up. envd's HTTP listener is the specific thing missing.

Suspected mechanism

packages/envd/main.go run() orders startup like this:

1. go host.PollForMMDSOpts(ctx, mmdsChan, defaults.EnvVars) — MMDS goroutine is the first thing spawned.
2. A chain of synchronous initializers: chi.NewRouter, filesystemRpc.Handle, createCgroupManager(), cgroups.NewWorkloadFreezer(...), processRpc.Handle, optional resumeHandover block, api.New, publicport.NewScanner, publicport.NewForwarder, portForwarder.StartForwarding(ctx) (go), portScanner.ScanAndBroadcast() (go).
3. Finally s.ListenAndServe().

If any of the synchronous steps in (2) blocks or panics into a swallowed / recovered path, the MMDS goroutine keeps ticking while ListenAndServe is never reached. Externally the process looks fully alive: MMDS side effects visible, /proc entry present, RSS non-zero, no exit — but no one is listening on 49983.

We could not grab an in-VM goroutine dump on a live stuck instance because getting a shell into the VM depends on envd's exec path, which is exactly what's broken. pprof served over HTTP needs the very listener that's missing. A vsock- or MMDS-mediated diagnostic channel would be the only way to surface it — mentioning this partly to ask whether the project would accept such a probe.

Suggested mitigations (either would independently avoid the observable state)

1. Listener-first startup. Move net.Listen("tcp", ":<port>") before MMDS poll / port scanner / cgroup manager, and only start those background goroutines once the socket is bound (or fail fast). Then wrap the post-listener init in bounded-timeout goroutines so a hang there surfaces as a listener that accepts + returns 5xx, rather than a listener that never binds.

2. Phased startup logging. At minimum, log each phase of run() before entering it (envd startup: opening cgroup manager etc.), and turn every recovered-panic path in run() into an os.Exit(1). That converts the silent variant into a visible crash, which the supervisor can restart and the operator can see in /var/log.

3. Readiness contract for consumers. Document that anything driving envd (orchestrator, snapshot resume, self-hosted forks) MUST TCP-dial the envd port before marking a sandbox usable, not just wait on the Firecracker socket. Every self-host consumer currently has to reinvent this.

Repro

Not reduced to a minimal repro yet — it's a rare startup-race pattern. Happens across multiple kernel versions, multiple templates, multiple hosts. Happy to gather additional data (in-VM strace of envd, MMDS packet counts, guest ss -tlnp at boot) against any specific hypothesis you'd like tested.

Start in packages/envd/main.go, especially run(), and trace the synchronous initializers between the MMDS poller and s.ListenAndServe(). Compare the healthy and stuck startup observations, then narrow the rare race with targeted boot diagnostics; done means the cause is identified and envd either binds its listener or exits visibly so the supervisor can restart it.