cloudflare / cloudflare/workerd
🐛 Bug Report: `std::terminate() called with no exception` when a tail consumer's target process dies abruptly (Windows)
- Dominant language
- C++
- Stars
- 8.7k
- Forks
- 739
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 174
Description
## Summary
On Windows, a workerd process aborts with
```
*** std::terminate() called with no exception
```
when a worker in that process declares a `tailConsumer` whose target lives in a **second workerd process** (reached via an external service over a loopback socket), and that second process **dies abruptly** (`SIGKILL` / `taskkill /T /F`).
There is no exception, no JS stack, and the abort happens on a non-main thread. The process dies immediately rather than surfacing a delivery error for the tail event.
## Version / platform
- workerd `1.20260731.1`
- `compatibility_date` `2025-05-01`
- **Windows only.** Not reproducible on macOS or Linux across many hundreds of cycles.
- Reached via Miniflare (`workers-sdk`), which is what wires the cross-process tail consumer up; see "How it's configured" below.
## Signature
```
*** std::terminate() called with no exception
stack: 7ffd4a141249 7ff692d863a5 7ffd4a15baff 7ffd4a143a9c 7ffd4a15ae8c 7ffd4a142ce1 7ffd4a15b8f7
7ffd583a700e 7ffd5824e592 7ffd4a15b3d5 7ffd4a141c1f 7ffd4a142dd7 7ffd4a15b8f7 7ffd583a6f8e
7ffd58252326 7ffd5824a960 7ffd555a75f9 7ffd4a1455a8 7ff69441045f 7ff692d85f63 7ff694415512
7ff69441c416 7ff694415a6b 7ff69441c416 7ff694412fbb 7ff692d71e77 7ff69624219e 7ffd56a1e8d6
7ffd582cc53b
```
The stack is **byte-identical across every occurrence** once ASLR base addresses are normalised, so this is a single deterministic path. We could not symbolize it — `$LLVM_SYMBOLIZER` is unset on the CI runners and the release binary is stripped — which is the main reason we're filing rather than proposing a fix.
## How it's configured
Two independent workerd processes, each with its own config, cross-referencing each other:
- **Process 1** hosts worker `A`, which has `assets` configured (so its incoming requests are fronted by an asset/router worker pair and its user worker is reached over RPC).
- **Process 2** hosts worker `B`, which declares `tailConsumers: [A]`.
`B`'s tail consumer is not a local service — it's an external service that proxies over a loopback socket into process 1.
When **process 1 dies abruptly**, process 2 aborts with the signature above.
Notably the abort also happens when both sessions are **completely idle** — no request ever traverses the binding. So this is not triggered by an in-flight tail event from live traffic; merely having the consumer configured and the peer die is enough.
## Reproduction rate and discriminators
Measured on Windows CI, isolating one variable at a time (each "arm" is repeated kill/restart cycles against an otherwise identical setup):
| Arm | Crash rate |
| --- | --- |
| `B` tails `A`, where `A` has **assets** | **4/4** (and 12/13 in an earlier round) |
| `B` has **no** tail consumer | 0/4 (16 cycles) |
| `B` tails a peer **without assets** | 0/4 (18 cycles) |
| `B` tails `A` (assets), service binding between them **removed** | **4/4** |
So:
- The tail consumer edge is **necessary**.
- The peer having **assets** configured is **necessary**.
- A service binding between the two is **irrelevant** — removing it does not help, and the crash occurs with no traffic at all.
- Killing an assets-enabled worker that is *not* anyone's tail consumer: 0/6 (28 cycles). Killing a non-assets worker that *is* bound as a service: 0/6 (23 cycles).
## What we ruled out
- **Traffic over the binding** — idle sessions still crash.
- **Process restarts by themselves** — 0/3 with 7–10 restarts and no peer present.
- **Registry/config churn with no live peer** — 0/3.
- **Peer count, port contention, start order** — 0/41 combined.
- **CPU starvation** — 0/3 with a live peer under load.
- **Reporting tail-forward failures at the JS layer** ([workers-sdk#14993](https://github.com/cloudflare/workers-sdk/pull/14993)) — 8/8 still crashed, i.e. the abort is below the JS-visible layer.
- **Bypassing the extra RPC hop** that assets introduces for tail delivery — 4/4 and 3/4 still crashed. This is worth emphasising because it's the obvious explanation for why `assets` is a discriminator, and it appears to be wrong. We don't have a mechanism that explains the `assets` dependency.
## Impact
This is the direct cause of a persistent CI flake in `workers-sdk` (multi-session local dev tests), where abruptly terminating one `vite dev` / `wrangler dev` session kills a sibling session's runtime. For users running several local dev sessions with tail consumers pointed at each other, stopping one session can take down another.
The `workers-sdk` side already surfaces the crash report and restarts the runtime ([workers-sdk#14989](https://github.com/cloudflare/workers-sdk/pull/14989)), so it's recoverable — but the abort itself looks like it should be a delivery error rather than a process kill.
## What would help
Even without a fix: if a maintainer recognises this stack shape, or can point at where tail-event delivery to an unreachable external service could reach `std::terminate` on a non-main thread, we can narrow it from our side. We're happy to run instrumented builds on Windows CI — that's where we have a reliable ~100% repro.
Contributor guide
Research direction
Start by reproducing the Windows-only failure with two workerd processes, a tailConsumer, and an abruptly terminated assets-enabled peer. Trace external tail-event delivery and peer teardown using an instrumented build and the normalized stack; done means the unreachable peer produces a delivery error without calling std::terminate or killing the sibling process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100