cloudflare / cloudflare/cloudflare-os
executeCode: every binding call fails with "Subrequest depth limit exceeded" on production since #177
- Dominant language
- TypeScript
- Stars
- 9.9k
- Forks
- 1.2k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 107
Description
Since #177, on a real Cloudflare production deployment, the **first outbound call from `executeCode` to any binding** fails:
```
Error: Subrequest depth limit exceeded. This request recursed through Workers too many times.
at async default (agent.js:2:19)
```
This is not specific to one gatekeeper: it hits gatekeeper bindings and gadget RPC (`env..()`) alike, which makes `executeCode` unusable for anything that touches the outside world. Intermittently the same calls surface as `Timed out waiting for logs from code execution` instead, which I suspect is the same failure arriving through a path that can't report it.
## Repro
The entire `executeCode` block — nothing else in it:
```js
export default async function(self, env, ctx) {
const actives = await env.SCHEDULER.list();
console.log(JSON.stringify(actives, null, 1));
}
```
`console.log` from the same invocation runs fine when placed before the call, so the module loads and starts up correctly. What dies is the first outbound call.
## Regression window
Works at `213ea6a`. Broken at `dd2b015`. Same workspace, same chat, same code, ~1 hour apart; the only thing that changed in between was the deployed backend. Before the upgrade the same session had made dozens of binding calls (BigQuery reads through a gatekeeper, gadget RPC, `SCHEDULER.calendarAt`) without a single incident.
## Root cause
#177 changed how the code-mode worker is loaded in `overseer.ts`:
```diff
- let restoreGadgetId = this.executeCodeRestoreTarget();
- if (restoreGadgetId === undefined) {
- entrypoint = this.env.LOADER.load(workerDef).getEntrypoint();
- } else {
- // Wacky hack: Load the code mode dynamic worker through `ctx.restore()` [...]
- let codeId = crypto.randomUUID();
- try {
- this.#codeIdMap.set(codeId, workerDef);
- entrypoint = await this.ctx.restore({
- type: "gadget",
- gadgetId: restoreGadgetId,
- codeId,
- });
- } finally {
- this.#codeIdMap.delete(codeId);
- }
- }
+ let entrypoint = this.env.LOADER.load(workerDef).getEntrypoint();
```
Loading the worker directly with `LOADER.load()` from the in-flight request appears to leave it one level deeper in the subrequest chain than the `ctx.restore()` path did. A binding call from executed code already traverses several Workers — executeCode worker → `GatekeeperLoopback` → Overseer DO → gatekeeper Worker → and back into the Overseer for `authorizeObservation()` — so one extra level is enough to cross the limit on the first call.
I can't verify the runtime semantics of `ctx.restore()` vs `LOADER.load()` from outside, so treat that last paragraph as the mechanism I inferred rather than something I confirmed. The causality itself is solid, though — see below.
## Why this wouldn't show up in CI or local dev
The "recursed through Workers too many times" cap is enforced by the production platform. `wrangler dev` / local workerd don't apply it the same way, so a deployment to real Cloudflare is the first place it can bite.
## Confirmed by the workaround
We patched our deployment to restore the pre-#177 load path — keeping the direct `LOADER.load()` for the no-gadget case exactly as before — and changed nothing else. The same minimal block above went straight from failing to returning `[]`, and binding calls work normally again. Same environment, same call, single-variable change.
The forging added by #177 is unaffected by that revert: `env.[restore]()` goes through `forgeRestoreStubForBinding()`, which loads its own worker under its own `codeId`.
I have that as a ~20-line patch and I'm happy to open it as a PR, but I'd rather not presume — you removed that `ctx.restore()` deliberately, and you may prefer to keep `LOADER.load()` and cut depth somewhere else in the chain. Just say which you'd prefer.
## Related, minor
With several `env.[restore]()` calls in a **single** `executeCode` invocation, `#soleForgedRestoreTarget()` returns `undefined` (because `targets.size > 1`) and `bindHook()` attributes every resulting hook to the workspace's first gadget. The callbacks themselves target the right gadgets — your comment says as much ("for bookkeeping; the callback itself already encapsulates the correct restore target") — but since the Connections tab filters hooks by gadget, the hooks become unreachable from the dashboard they belong to, and there's no error to hint at it. Registering one per invocation works fine. This is the TODO already sitting next to that code; happy to file it separately if you'd like it tracked.
## Deployment context
Self-hosted production deployment on a customer Cloudflare account: 8 Workers (Workshop + 7 gatekeepers, some of them our own) wired through service bindings, Workshop behind Cloudflare Access, no AI Gateway. Glad to provide more detail or test a fix.
Contributor guide
Research direction
Read `overseer.ts` around the code-mode worker loading change in #177 and trace how `executeCode` reaches a binding. The issue provides a minimal reproduction using `env.SCHEDULER.list()` and reports that reverting the loading path fixes it in production; no test file is named. Done means outbound binding calls from `executeCode` work on a real Cloudflare deployment without exceeding the subrequest depth limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100