CopilotKit / CopilotKit/outpost

worker: shutdown() does not actually drain in-flight jobs

Open
#181 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: infrastructure roadmap roadmap: now
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

Found during the four-round review of #180. Not fixed there — apps/worker/src/index.ts was deliberately left at main's behaviour.

Worker.start() registers its own SIGTERM/SIGINT handlers (packages/outpost/queue/src/worker.ts) that call this.stop(). Those are registered before index.ts's handlers, so on SIGTERM:

  1. The Worker's own handler runs first. stop() synchronously sets running = false, then begins awaiting the drain — nobody awaits that promise.
  2. index.ts's shutdown() then calls await worker.stop(), which hits if (!this.running) return and returns immediately.
  3. shutdown() proceeds to prisma.$disconnect() and process.exit(0) while jobs are still executing.

So the drain Worker.stop() exists to provide never happens on a real shutdown. In-flight AI_RESPONSE (120s) and HUBSPOT_SYNC/ACCOUNT_SCORING (300s) jobs are killed mid-execution with Prisma disconnected under them.

This compounds with a second finding: nothing in the queue reclaims a job stuck in PROCESSING. worker.ts sets status='PROCESSING', lockedAt=NOW() on claim and only clears lockedAt on completion/failure/retry; there is no stale-lock reaper in job-cleanup.ts or anywhere else. So any hard exit mid-job strands those rows permanently, not until the next boot.

Suggested fix: a registerSignalHandlers: false option on Worker so the app owns shutdown, or make stop() joinable (cache the in-flight stop promise and return it rather than early-returning on !running). A stale-lock reaper is worth its own consideration either way.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with apps/worker/src/index.ts and packages/outpost/queue/src/worker.ts, then inspect job-cleanup.ts for existing cleanup behavior. Trace both signal-handler paths and the PROCESSING lock lifecycle; done means shutdown waits for in-flight jobs before disconnecting, with the stale-lock handling decision addressed or explicitly separated.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.