UserWorker.fetch never settles when a request with a body is rejected before dispatch (WorkerAlreadyRetired)

Open
#742 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
deno, javascript, rust, typescript
Domain
api, backend

Research direction

Start in ext/workers/user_workers.js by tracing UserWorker.fetch(), requestBodyPromise, and responsePromise, then run the supplied repro with the Docker command and POST curl request. Verify that a rejected send with a request body settles and surfaces WorkerAlreadyRetired, allowing the main service to retry with a fresh worker as described.

Written by the indexing model from the issue text.

Description

Versions: edge-runtime v1.74.3, the image used by Supabase CLI 2.114.0 functions serve. UserWorker.fetch() in ext/workers/user_workers.js is unchanged in v1.76.2.

Symptom. When op_user_worker_fetch_send rejects a request, for example with WorkerAlreadyRetired after the selected worker retired between userWorkers.create() and fetch(), the outcome depends on the body:

  • a request without a body rejects promptly;
  • a request with a body never settles, so the main service can neither answer nor retry it.

In both cases the runtime logs user worker failed to respond: request cannot be handled because the worker has already retired. With a body, the client then hangs until its gateway times out.

Likely cause (from reading ext/workers/user_workers.js). fetch() awaits Promise.allSettled([requestBodyPromise, responsePromise]). requestBodyPromise pipes the body into requestBodyRid. After a rejected send no worker reads that resource, so the pipe apparently never completes and allSettled never resolves.

Reproduction (edge-runtime v1.74.3). The main service is the one from supabase/cli#6675, which reproduces the CLI side of the same race:

repro/main/index.ts and repro/fn/index.ts
// repro/main/index.ts
// Deterministic stand-in for the CLI bootstrap's race: /slow creates a worker
// (4 s wall clock, retired at ~2 s) and keeps it busy. The first /fast is
// dispatched to that same worker object, as if `create()` had returned it just
// before retirement, and gets a real WorkerAlreadyRetired. Every later /fast
// calls `create()` like the CLI does, which hands out a fresh worker. Errors
// are answered with the CLI 2.114.0 bootstrap fallback, byte for byte.
let worker: any;
let staleDispatchUsed = false;
const options = {
  servicePath: '/work/fn', memoryLimitMb: 64, workerTimeoutMs: 4000, noModuleCache: false,
  envVars: [], forceCreate: false, cpuTimeSoftLimitMs: 10000, cpuTimeHardLimitMs: 20000,
};
function prepare(req: Request) {
  const clone = new Request(new URL(req.url), req.clone());
  EdgeRuntime.applySupabaseTag(req, clone);
  return clone;
}
Deno.serve(async (req) => {
  if (new URL(req.url).pathname === '/health') return new Response('ok');
  try {
    if (new URL(req.url).pathname.endsWith('/slow')) {
      worker = await EdgeRuntime.userWorkers.create(options);
      return await worker.fetch(prepare(req));
    }
    const target = staleDispatchUsed ? await EdgeRuntime.userWorkers.create(options) : worker;
    staleDispatchUsed = true;
    return await target.fetch(prepare(req));
  } catch (p) {
    console.error(p);
    return new Response(JSON.stringify({ code: 'Internal Server Error', message: 'Request failed due to an internal server error', trace: JSON.stringify(p.stack) }), { status: 500, headers: { 'Content-Type': 'application/json' } });
  }
});
// repro/fn/index.ts
Deno.serve(async (req) => {
  if (new URL(req.url).pathname.endsWith('/slow')) await new Promise((r) => setTimeout(r, 3000));
  return new Response('fn-ok');
});
docker run --rm -v "$PWD/repro:/work:ro" -p 127.0.0.1:18081:8081 --entrypoint edge-runtime \
  ghcr.io/supabase/edge-runtime:v1.74.3 start --main-service=/work/main --port=8081

In a second terminal, once curl -s http://127.0.0.1:18081/health answers ok, run:

curl -s http://127.0.0.1:18081/slow &   # creates the worker (4 s wall clock) and keeps it busy
sleep 2.4; curl -s --max-time 8 -X POST -H 'content-type: application/json' -d '{"a":1}' http://127.0.0.1:18081/fast

As a GET, the same /fast request returns the WorkerAlreadyRetired error at once (about 25 ms). As a POST it gets no response: curl gives up after 8 s, although the runtime has already logged the rejection.

Suggested fix (untested sketch). Stop piping the body once the send has failed:

const requestBodyAbort = new AbortController();
if (hasBody) {
  requestBodyPromise = body.pipeTo(writableStreamForRid(requestBodyRid), {
    signal: signal ? AbortSignal.any([signal, requestBodyAbort.signal]) : requestBodyAbort.signal,
  });
}
const responsePromise = op_user_worker_fetch_send(this.key, requestRid, requestBodyRid, tag.streamRid, tag.watcherRid)
  .catch((error) => {
    // The request never reached a worker; nobody will read its body.
    requestBodyAbort.abort(error);
    throw error;
  });

An alternative is for the Rust side to close the request-body resource when send_request fails. Either way, WorkerAlreadyRetired then surfaces for every request, and a main service can replay it with a fresh worker, as examples/main/index.ts does.

Dominant language
Rust
Stars
984
Forks
125
Avg merge
1d 1h
Merged PRs (30d)
5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from supabase/edge-runtime

All issues in supabase/edge-runtime

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.