UserWorker.fetch never settles when a request with a body is rejected before dispatch (WorkerAlreadyRetired)
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- deno, javascript, rust, typescript
Research direction
Start in ext/workers/user_workers.js by tracing UserWorker.fetch(), requestBodyPromise, and responsePromise, then run the supplied repro with the Docker command and POST curl request. Verify that a rejected send with a request body settles and surfaces WorkerAlreadyRetired, allowing the main service to retry with a fresh worker as described.
Written by the indexing model from the issue text.
Description
Versions: edge-runtime v1.74.3, the image used by Supabase CLI 2.114.0 functions serve. UserWorker.fetch() in ext/workers/user_workers.js is unchanged in v1.76.2.
Symptom. When op_user_worker_fetch_send rejects a request, for example with WorkerAlreadyRetired after the selected worker retired between userWorkers.create() and fetch(), the outcome depends on the body:
- a request without a body rejects promptly;
- a request with a body never settles, so the main service can neither answer nor retry it.
In both cases the runtime logs user worker failed to respond: request cannot be handled because the worker has already retired. With a body, the client then hangs until its gateway times out.
Likely cause (from reading ext/workers/user_workers.js). fetch() awaits Promise.allSettled([requestBodyPromise, responsePromise]). requestBodyPromise pipes the body into requestBodyRid. After a rejected send no worker reads that resource, so the pipe apparently never completes and allSettled never resolves.
Reproduction (edge-runtime v1.74.3). The main service is the one from supabase/cli#6675, which reproduces the CLI side of the same race:
repro/main/index.ts and repro/fn/index.ts
// repro/main/index.ts
// Deterministic stand-in for the CLI bootstrap's race: /slow creates a worker
// (4 s wall clock, retired at ~2 s) and keeps it busy. The first /fast is
// dispatched to that same worker object, as if `create()` had returned it just
// before retirement, and gets a real WorkerAlreadyRetired. Every later /fast
// calls `create()` like the CLI does, which hands out a fresh worker. Errors
// are answered with the CLI 2.114.0 bootstrap fallback, byte for byte.
let worker: any;
let staleDispatchUsed = false;
const options = {
servicePath: '/work/fn', memoryLimitMb: 64, workerTimeoutMs: 4000, noModuleCache: false,
envVars: [], forceCreate: false, cpuTimeSoftLimitMs: 10000, cpuTimeHardLimitMs: 20000,
};
function prepare(req: Request) {
const clone = new Request(new URL(req.url), req.clone());
EdgeRuntime.applySupabaseTag(req, clone);
return clone;
}
Deno.serve(async (req) => {
if (new URL(req.url).pathname === '/health') return new Response('ok');
try {
if (new URL(req.url).pathname.endsWith('/slow')) {
worker = await EdgeRuntime.userWorkers.create(options);
return await worker.fetch(prepare(req));
}
const target = staleDispatchUsed ? await EdgeRuntime.userWorkers.create(options) : worker;
staleDispatchUsed = true;
return await target.fetch(prepare(req));
} catch (p) {
console.error(p);
return new Response(JSON.stringify({ code: 'Internal Server Error', message: 'Request failed due to an internal server error', trace: JSON.stringify(p.stack) }), { status: 500, headers: { 'Content-Type': 'application/json' } });
}
});
// repro/fn/index.ts
Deno.serve(async (req) => {
if (new URL(req.url).pathname.endsWith('/slow')) await new Promise((r) => setTimeout(r, 3000));
return new Response('fn-ok');
});
docker run --rm -v "$PWD/repro:/work:ro" -p 127.0.0.1:18081:8081 --entrypoint edge-runtime \
ghcr.io/supabase/edge-runtime:v1.74.3 start --main-service=/work/main --port=8081
In a second terminal, once curl -s http://127.0.0.1:18081/health answers ok, run:
curl -s http://127.0.0.1:18081/slow & # creates the worker (4 s wall clock) and keeps it busy
sleep 2.4; curl -s --max-time 8 -X POST -H 'content-type: application/json' -d '{"a":1}' http://127.0.0.1:18081/fast
As a GET, the same /fast request returns the WorkerAlreadyRetired error at once (about 25 ms). As a POST it gets no response: curl gives up after 8 s, although the runtime has already logged the rejection.
Suggested fix (untested sketch). Stop piping the body once the send has failed:
const requestBodyAbort = new AbortController();
if (hasBody) {
requestBodyPromise = body.pipeTo(writableStreamForRid(requestBodyRid), {
signal: signal ? AbortSignal.any([signal, requestBodyAbort.signal]) : requestBodyAbort.signal,
});
}
const responsePromise = op_user_worker_fetch_send(this.key, requestRid, requestBodyRid, tag.streamRid, tag.watcherRid)
.catch((error) => {
// The request never reached a worker; nobody will read its body.
requestBodyAbort.abort(error);
throw error;
});
An alternative is for the Rust side to close the request-body resource when send_request fails. Either way, WorkerAlreadyRetired then surfaces for every request, and a main service can replay it with a fresh worker, as examples/main/index.ts does.
- Dominant language
- Rust
- Stars
- 984
- Forks
- 125
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from supabase/edge-runtime
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
supabase/edge-runtime#741 ·
-
Difficulty 5/5 Over a week Newbie friendliness 42/100
supabase/edge-runtime#740 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
supabase/edge-runtime#738 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
supabase/edge-runtime#737 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
supabase/edge-runtime#739 · 2 comments · 1 reaction ·
All issues in supabase/edge-runtime
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
kwakseongjae/auto-hwp#319 ·
-
area:cli bug filter-quality good first issue priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
bevyengine/bevy#25861 ·
-
comp-datalake
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
ClickHouse/ClickHouse#121222 ·
-
enhancement remote
Difficulty 2/5 1-3 hours Newbie friendliness 68/100