cloudflare / cloudflare/workerd
Unbounded native memory growth from repeated cross-worker fetch() subrequests (uncollected receiver-isolate garbage)
- Dominant language
- C++
- Stars
- 8.7k
- Forks
- 739
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 174
Description
## Summary
Under a sustained stream of cross-worker `fetch()` **subrequests**, `workerd`'s RSS grows
without bound and the process is eventually OOM-killed. The growth is **native (C++)
memory** — per-subrequest object graphs in the **receiver** isolate whose V8 wrappers are
never collected, because that isolate's JS heap stays tiny and the native weight does not
appear to be reported to V8, so V8 never schedules a GC.
It is **uncollected garbage, not a true leak**: forcing a GC on the receiver isolate frees
all of it and the memory is reused.
This surfaced originally as an OOM in `@cloudflare/vite-plugin` dev servers
([workers-sdk#14701](https://github.com/cloudflare/workers-sdk/issues/14701)), but it
reproduces in plain `workerd` with two trivial Workers and no wrangler / vite / miniflare.
## Reproduction (dependency-free)
Three files + a config. Worker `main` issues N `fetch()` subrequests to worker `target`.
`config.capnp`:
```capnp
using Workerd = import "/workerd/workerd.capnp";
const config :Workerd.Config = (
services = [
(name = "main", worker = .mainWorker),
(name = "target", worker = .targetWorker),
],
sockets = [
(name = "http", address = "*:8080", http = (), service = "main"),
],
);
const mainWorker :Workerd.Worker = (
modules = [ (name = "main.js", esModule = embed "main.js") ],
bindings = [ (name = "TARGET", service = "target") ],
compatibilityDate = "2024-12-30",
);
const targetWorker :Workerd.Worker = (
modules = [ (name = "target.js", esModule = embed "target.js") ],
compatibilityDate = "2024-12-30",
);
```
`main.js`:
```js
export default {
async fetch(request, env) {
const count = Number(new URL(request.url).searchParams.get("n") ?? "100");
for (let i = 0; i < count; i++) {
const res = await env.TARGET.fetch("http://target/");
await res.text();
}
return new Response(`did ${count} subrequests`);
},
};
```
`target.js`:
```js
export default {
async fetch(request) {
return new Response("ok");
},
};
```
Run it and drive load:
```sh
workerd serve --inspector-addr=localhost:9230 config.capnp
# in another terminal — 20,000 subrequests total:
for i in $(seq 1 200); do curl -s "http://localhost:8080/?n=100" >/dev/null; done
# sample RSS:
ps -o rss= -p "$(pgrep -f 'workerd serve')" | awk '{printf "%d MB\n", $1/1024}'
```
## Observed (`workerd` 2026-07-21, macOS arm64)
RSS grows linearly and never comes back down on its own:
| subrequests | RSS | live `DeleteQueue` (proxy for per-request graphs) |
| ----------: | -----: | ------------------------------------------------: |
| 0 | 28 MB | 2 |
| 4,000 | 79 MB | 4,042 |
| 8,000 | 121 MB | 8,082 |
| 12,000 | 165 MB | 12,122 |
| 16,000 | 201 MB | 16,162 |
| 20,000 | 224 MB | 20,202 |
≈ one leaked per-request object graph per subrequest. `DeleteQueue` was counted with
macOS `heap --addresses=workerd::DeleteQueue --noContent ` and is used only as an
easy-to-count proxy for the number of un-GC'd per-subrequest graphs (it is not itself the
bulk of the memory).
## The memory is reclaimable garbage
`HeapProfiler.takeHeapSnapshot` forces a full mark-compact GC. Applying it via CDP after
20,000 subrequests:
- GC the **receiver** (`target`) isolate → `DeleteQueue` → **0**, RSS returns toward
baseline, and the freed memory is reused by subsequent load.
- GC the **caller** (`main`) isolate → no effect.
So the retained objects are unreachable garbage in the receiver isolate; V8 simply never
schedules a GC there under sustained subrequest load.
The same behaviour was confirmed in a real multi-isolate app (the `@cloudflare/vite-plugin`
dev server, ~21 isolates): after driving HMR load the live `DeleteQueue` count was ~5,700,
and forcing GC specifically on the isolate that *receives* the module-fetch subrequests
dropped it to ~87 while every other isolate's GC did nothing. The catch that made this hard
to see: that receiver isolate's inspector debug path contains a literal `#`, which
URL-based WebSocket clients (and Chrome DevTools) silently truncate as a fragment, so a
naive "force GC via DevTools" always lands on the wrong (caller) isolate and appears to do
nothing.
## What is allocated per subrequest (root cause)
Each `env.TARGET.fetch(...)` sets up an `IoContext` in the receiver isolate and runs the
entrypoint. Besides the expected `Request`/`Response`/`Headers`/`ReadableStream`, each
invocation allocates:
- **Two trace-span async-context frames.** `worker-entrypoint.c++` builds both a
`traceScope` and a `userTraceScope` per `fetch` (`worker-entrypoint.c++:449-451`, also
`:671-672`, `:790-791`, `:864-866`, `:962-963`), via
`IoContext::makeAsyncTraceScope` (`io-context.c++:1136`) and
`IoContext::makeUserAsyncTraceScope` (`io-context.c++:1162`). Each heap-allocates a
`SpanParent`, wraps it in an `IoOwn` (`addObject`, `io-context.h:702`), wraps that in an
opaque V8 handle (`jsg::wrapOpaque`), and builds an `AsyncContextFrame::StorageScope`
holding a `v8Ref`. `makeAsyncTraceScope` short-circuits the *span* when tracing is off
(`io-context.c++:1148`) but still performs the wrapper allocations — there's an explicit
`TODO(cleanup)` about this at `io-context.c++:1145-1146`; `makeUserAsyncTraceScope` has
no short-circuit at all.
- **A promise-context tag holding a `DeleteQueue` ref.** `runInContextScope` installs a
`v8::Isolate::PromiseContextScope` via `getPromiseContextTag`
(`io-context.c++:1287-1288`), which creates an `IoCrossContextExecutor` holding
`deleteQueue.queue.addRef()` (`io-context.c++:1652-1657`, `io-own.h:124-126`), stored in
an opaque V8 object owned by the `IoContext`. One `DeleteQueue` per subrequest.
After the subrequest completes the `IoContext` is torn down and all of the above become
unreachable, but they are **V8-wrapped native objects**, so their C++ memory is only
released when V8 collects the wrappers in the receiver isolate. The receiver isolate's JS
heap stays ~17-22 MB and the native weight is apparently not reported to V8, so V8's
JS-heap-growth-driven GC scheduler never fires under sustained load → unbounded RSS.
## Possible directions
1. Report the native memory behind these per-request wrappers to V8 (external-memory
accounting) so GC is scheduled under subrequest load.
2. Elide the per-invocation trace-scope allocations when tracing is disabled — the
`TODO(cleanup)` at `io-context.c++:1145-1146`, extended to `makeUserAsyncTraceScope`.
3. Prompt GC on idle receiver isolates that have accumulated many completed `IoContext`s.
## Ruled out
- **Vite / JS retention** — heap snapshots of all inspectable isolates are flat.
- **Durable Objects** — reproduces with no DO; only one `IoContext` is live after the load.
- **Cross-request promise resolution** — toggling
`no_handle_cross_request_promise_resolution` had zero effect (it only gates a
`SetPromiseCrossContextResolveCallback`, not the promise-context tag / `DeleteQueue`).
## Environment
- `workerd` 2026-07-21 (`@cloudflare/workerd-darwin-arm64@1.20260721.1`)
- macOS, arm64
- Line numbers above are from the `workerd` `main` branch around that date.
Contributor guide
Research direction
Reproduce the growth with the three worker files and config.capnp, then inspect worker-entrypoint.c++, io-context.c++, io-context.h, and io-own.h at the cited trace-scope and promise-context locations. Compare the listed accounting, allocation-elision, and idle-GC directions; done means sustained cross-worker fetch() load no longer causes unbounded RSS growth and receiver-isolate GC reclaims the per-request graphs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, javascript
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100