CopilotKit / CopilotKit/outpost
worker: shutdown() does not actually drain in-flight jobs
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
Found during the four-round review of #180. Not fixed there — apps/worker/src/index.ts was deliberately left at main's behaviour.
Worker.start() registers its own SIGTERM/SIGINT handlers (packages/outpost/queue/src/worker.ts) that call this.stop(). Those are registered before index.ts's handlers, so on SIGTERM:
- The Worker's own handler runs first.
stop()synchronously setsrunning = false, then begins awaiting the drain — nobody awaits that promise. index.ts'sshutdown()then callsawait worker.stop(), which hitsif (!this.running) returnand returns immediately.shutdown()proceeds toprisma.$disconnect()andprocess.exit(0)while jobs are still executing.
So the drain Worker.stop() exists to provide never happens on a real shutdown. In-flight AI_RESPONSE (120s) and HUBSPOT_SYNC/ACCOUNT_SCORING (300s) jobs are killed mid-execution with Prisma disconnected under them.
This compounds with a second finding: nothing in the queue reclaims a job stuck in PROCESSING. worker.ts sets status='PROCESSING', lockedAt=NOW() on claim and only clears lockedAt on completion/failure/retry; there is no stale-lock reaper in job-cleanup.ts or anywhere else. So any hard exit mid-job strands those rows permanently, not until the next boot.
Suggested fix: a registerSignalHandlers: false option on Worker so the app owns shutdown, or make stop() joinable (cache the in-flight stop promise and return it rather than early-returning on !running). A stale-lock reaper is worth its own consideration either way.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with apps/worker/src/index.ts and packages/outpost/queue/src/worker.ts, then inspect job-cleanup.ts for existing cleanup behavior. Trace both signal-handler paths and the PROCESSING lock lifecycle; done means shutdown waits for in-flight jobs before disconnecting, with the stale-lock handling decision addressed or explicitly separated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100