cloudflare / cloudflare/workerd

An isolate stays in rotation after an invocation is terminated for exceeding resource limits, so any stranded module-scope state poisons every later request

Open
#7,276 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

## Summary

When a Worker invocation is terminated for exceeding resource limits, the isolate is **retained and keeps receiving new requests**, but the terminated invocation's frames are never unwound. Any `try/finally` in flight does not run. If the abandoned frame held a mutation on module-scope state — which is normal for framework caches that intentionally live across requests — the isolate is left permanently broken while still being routed traffic.

The result is a large amplification: a low rate of resource-limit kills becomes a much higher rate of user-visible failures, because each kill can convert one isolate into a persistent failure source for minutes.

## Measured behaviour

Cloudflare Pages, Angular SSR Function, strictly sequential requests (concurrency 1, so this is not request overlap), every URL cache-busted. Per-isolate counters are stamped on each response by the Function itself; `abandoned` is `starts - ends - 1`, where `ends` is incremented in a `finally`.

| | |
|---|---|
| Requests 1–24 on isolate `A` | rendered normally, ~700 ms each |
| 2 requests | `503 Worker exceeded resource limits` (Cloudflare's own error page, our code never returned) |
| `abandoned` on isolate `A` | `2` — those two invocations began and never finished |
| Requests 27–78 on isolate `A` | **52 consecutive application-level failures, zero recoveries** |

Reproduced repeatedly with runs of 52, 121, 169 and 290 consecutive failures on a single isolate. In one run the same isolate id was still being handed requests and still failing **9 minutes** after its first kill. Meanwhile it kept serving edge-cache hits perfectly — it could replay a response, just not produce one.

Two things follow that seem worth distinguishing:

1. `endRequest()` sits in a `finally` and still did not run, so termination is not surfaced to the isolate as an unwind at all.
2. The isolate remained eligible for routing indefinitely afterwards, with no signal available to it or to us that it was compromised.

## Why this is hard to handle in application code

We cannot repair the stranded state in our case (it is inside a framework's shared render cache, reachable by no public API, and clearing it by hand would race legitimate concurrent work). We also cannot ask for the isolate to be replaced: there is no API to signal "this isolate is unhealthy, stop routing to it" or to voluntarily terminate it.

So the only option left was to detect the condition and stop doing the work that fails — a circuit breaker that serves cached copies and skips the doomed code path. That contains the damage but leaves the isolate in rotation, degraded, until it is recycled on its own schedule.

## Requests, in rough order of usefulness

1. **Do not retain an isolate whose invocation was terminated for exceeding resource limits** — or at least prefer a fresh isolate for subsequent requests. Termination without unwinding means the isolate's invariants can no longer be assumed to hold.
2. **A way for a Worker to declare itself unhealthy** and be replaced (an explicit "retire this isolate" API), for cases where the Worker can detect the damage but not repair it.
3. **A signal that the previous invocation in this isolate was terminated** — e.g. something readable at the start of a request. Today we infer it from our own `starts`/`ends` gap, which works but is guesswork that every Worker author has to reinvent.
4. **Include the limit that was exceeded** (CPU vs memory) somewhere the account owner can see per-invocation. `pagesFunctionsInvocationsAdaptiveGroups.outcome` exists but is not reachable with a zone-scoped token, and the synthetic `503` carries no marker at all, so "which limit" is still unresolved for us.

## Related

Filed alongside https://github.com/angular/angular/issues/70614. The failure genuinely requires both halves: Angular keeps a re-entrancy flag on shared state whose reset depends on a `finally`, and the runtime terminates invocations without unwinding while keeping the isolate in service. Neither behaviour is unreasonable alone; together, one resource-limit kill is permanently fatal to a process that then keeps serving requests.

Happy to share the harness, per-isolate telemetry, or `wrangler tail` output.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported strictly sequential workload with per-isolate starts and ends counters, using the described resource-limit termination and subsequent requests. Trace the runtime's invocation termination and isolate-routing entry points; done should mean a terminated invocation cannot leave a compromised isolate serving requests, or that a clear replacement or unhealthy signal is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
angular, javascript
Domain
backend, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.