cloudflare / cloudflare/cloudflare-os

executeCode: every binding call fails with "Subrequest depth limit exceeded" on production since #177

Open
#282 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
9.9k
Forks
1.2k
Avg merge
1d 20h
Merged PRs (30d)
107

Description

Since #177, on a real Cloudflare production deployment, the **first outbound call from `executeCode` to any binding** fails:

```
Error: Subrequest depth limit exceeded. This request recursed through Workers too many times.
at async default (agent.js:2:19)
```

This is not specific to one gatekeeper: it hits gatekeeper bindings and gadget RPC (`env..()`) alike, which makes `executeCode` unusable for anything that touches the outside world. Intermittently the same calls surface as `Timed out waiting for logs from code execution` instead, which I suspect is the same failure arriving through a path that can't report it.

## Repro

The entire `executeCode` block — nothing else in it:

```js
export default async function(self, env, ctx) {
const actives = await env.SCHEDULER.list();
console.log(JSON.stringify(actives, null, 1));
}
```

`console.log` from the same invocation runs fine when placed before the call, so the module loads and starts up correctly. What dies is the first outbound call.

## Regression window

Works at `213ea6a`. Broken at `dd2b015`. Same workspace, same chat, same code, ~1 hour apart; the only thing that changed in between was the deployed backend. Before the upgrade the same session had made dozens of binding calls (BigQuery reads through a gatekeeper, gadget RPC, `SCHEDULER.calendarAt`) without a single incident.

## Root cause

#177 changed how the code-mode worker is loaded in `overseer.ts`:

```diff
- let restoreGadgetId = this.executeCodeRestoreTarget();
- if (restoreGadgetId === undefined) {
- entrypoint = this.env.LOADER.load(workerDef).getEntrypoint();
- } else {
- // Wacky hack: Load the code mode dynamic worker through `ctx.restore()` [...]
- let codeId = crypto.randomUUID();
- try {
- this.#codeIdMap.set(codeId, workerDef);
- entrypoint = await this.ctx.restore({
- type: "gadget",
- gadgetId: restoreGadgetId,
- codeId,
- });
- } finally {
- this.#codeIdMap.delete(codeId);
- }
- }
+ let entrypoint = this.env.LOADER.load(workerDef).getEntrypoint();
```

Loading the worker directly with `LOADER.load()` from the in-flight request appears to leave it one level deeper in the subrequest chain than the `ctx.restore()` path did. A binding call from executed code already traverses several Workers — executeCode worker → `GatekeeperLoopback` → Overseer DO → gatekeeper Worker → and back into the Overseer for `authorizeObservation()` — so one extra level is enough to cross the limit on the first call.

I can't verify the runtime semantics of `ctx.restore()` vs `LOADER.load()` from outside, so treat that last paragraph as the mechanism I inferred rather than something I confirmed. The causality itself is solid, though — see below.

## Why this wouldn't show up in CI or local dev

The "recursed through Workers too many times" cap is enforced by the production platform. `wrangler dev` / local workerd don't apply it the same way, so a deployment to real Cloudflare is the first place it can bite.

## Confirmed by the workaround

We patched our deployment to restore the pre-#177 load path — keeping the direct `LOADER.load()` for the no-gadget case exactly as before — and changed nothing else. The same minimal block above went straight from failing to returning `[]`, and binding calls work normally again. Same environment, same call, single-variable change.

The forging added by #177 is unaffected by that revert: `env.[restore]()` goes through `forgeRestoreStubForBinding()`, which loads its own worker under its own `codeId`.

I have that as a ~20-line patch and I'm happy to open it as a PR, but I'd rather not presume — you removed that `ctx.restore()` deliberately, and you may prefer to keep `LOADER.load()` and cut depth somewhere else in the chain. Just say which you'd prefer.

## Related, minor

With several `env.[restore]()` calls in a **single** `executeCode` invocation, `#soleForgedRestoreTarget()` returns `undefined` (because `targets.size > 1`) and `bindHook()` attributes every resulting hook to the workspace's first gadget. The callbacks themselves target the right gadgets — your comment says as much ("for bookkeeping; the callback itself already encapsulates the correct restore target") — but since the Connections tab filters hooks by gadget, the hooks become unreachable from the dashboard they belong to, and there's no error to hint at it. Registering one per invocation works fine. This is the TODO already sitting next to that code; happy to file it separately if you'd like it tracked.

## Deployment context

Self-hosted production deployment on a customer Cloudflare account: 8 Workers (Workshop + 7 gatekeepers, some of them our own) wired through service bindings, Workshop behind Cloudflare Access, no AI Gateway. Glad to provide more detail or test a fix.

Contributor guide

Open the contributing guide

Research direction

Read `overseer.ts` around the code-mode worker loading change in #177 and trace how `executeCode` reaches a binding. The issue provides a minimal reproduction using `env.SCHEDULER.list()` and reports that reverting the loading path fixes it in production; no test file is named. Done means outbound binding calls from `executeCode` work on a real Cloudflare deployment without exceeding the subrequest depth limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.