CopilotKit / CopilotKit/outpost

Keep a timed-out handler visible after runWithTimeout stops waiting for it

Open
#266 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: infrastructure roadmap: next
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

runWithTimeout in packages/outpost/queue/src/worker.ts is a Promise.race between the handler and a timer. The race settling does not cancel the handler, so on a timeout the job is marked failed, processJob's finally frees the slot and deletes the job's entry from activeJobStarts — and the handler keeps running.

That matters more since #262, because overdueJobCount is measured from activeJobStarts. An orphaned handler is removed from the map at the moment it becomes orphaned, so the count is zero by construction for exactly the work that escaped its timeout. The docstring on that field says a non-zero count means the timeout machinery failed; this is a failure of the timeout machinery that it cannot see.

What it looks like in production

The LLM provider stalls, with no AbortSignal on the pipeline's fetch. Every AI_RESPONSE job times out at 120s, handleFailure schedules a retry, and the poll immediately claims more. Each bad job leaves an orphan behind and is retried up to maxAttempts, so orphans accumulate — each holding a socket and a connection-pool checkout.

Steady state after ten minutes is a worker running noticeably more work than it believes it is, while /health answers 200 {"status":"ok"} and activeJobCount reports only whatever is currently raced. The eventual symptom is pool exhaustion surfacing as an unrelated Prisma error somewhere else entirely, which is a long way from the cause.

Shape of a fix

Keep the orphan visible rather than deleting it:

const started = this.activeJobStarts.get(job.id);
void promise.finally(() => this.orphanedHandlers.delete(job.id));
this.orphanedHandlers.set(job.id, started);

Publish orphanedHandlerCount on WorkerHealthStatus and answer 503 above a threshold — >= maxConcurrency is a defensible line, since past it the worker is running more work than it thinks it is. At minimum, console.error on every timeout naming the job id and type: today the only record is the console.warn inside handleFailure, which describes a retry rather than an abandoned handler.

The deeper fix is an AbortSignal threaded through the handler contract so a timeout actually stops the work. That is a larger change to a shared package and worth doing on its own.

Pre-existing on main; #262 neither introduced nor worsened it. Filed because #262's docs now reference the limit, and a documented limit with no tracking behind it tends to become a documented permanence.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/outpost/queue/src/worker.ts, tracing runWithTimeout, processJob, activeJobStarts, and the WorkerHealthStatus path. Compare the proposed orphan tracking and timeout logging with the existing health response and failure handling. Done means timed-out handlers remain observable, health reports orphanedHandlerCount, and the worker returns 503 at the agreed threshold.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.