CopilotKit / CopilotKit/outpost
worker: stop awaiting each claim batch, so concurrency limits and /health mean something
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
From the review of #224. These are the items that PR deliberately left out, and the first one is load-bearing for two others.
The batch await makes concurrency accounting dead code
poll() fully awaits Promise.allSettled over each claimed batch, and schedulePoll only runs after the poll resolves. So activeJobs.size and activeJobsByType are always 0 when availableSlots and typeLimit - activeForType are computed. Both expressions are unreachable: mutating each to ignore in-flight work passes the whole suite.
It does not over-fetch — the serialization accidentally bounds it — but per-type concurrency limits never bind, which is not what concurrencyByType looks like it does.
Which is also why /health is still wrong (#182, #184)
Because poll() blocks for the whole batch duration, lastPollTime stalls for as long as the longest job. buildHealthResponse compares it against STALE_POLL_MS = 60_000 with no active-job exemption, against a 120s AI_RESPONSE timeout. So a healthy worker mid-job returns 503 "stalled" and Railway restarts it — killing the job it was in the middle of.
Two ways out, and they should be decided together:
- Stop awaiting the batch (fixes the accounting too), or
- Exempt active jobs from the stale-poll check.
(1) is the better fix and makes (2) unnecessary. This is the reason /health has now been deferred three times; it should be one PR with the batch change.
An unregistered job type is terminally destroyed
FAILED with completedAt and no attempt increment — and job-cleanup.ts only deletes COMPLETED and DEAD_LETTER, so the rows accumulate forever. #224 added lockedAt: null, claimToken: null to that write, which means reclaim can't rescue them either.
Reachable only via claimAndProcessJobs, and production sets concurrencyByType so it takes the type-filtered path. Latent today; live the moment concurrencyByType is emptied or a worker ships without the full handler set.
A no-op release back to PENDING with no attempt consumed is the safe failure. Data-destroying path, so it wants its own change.
The timeout doesn't cancel the handler, and the drain has no deadline
runWithTimeout stops waiting for the handler; the handler keeps running. And AI_RESPONSE at 120s against Railway's ~30s SIGTERM grace means the graceful drain resolves well after SIGKILL already landed.
So the real shutdown path is still crash-abandonment plus reclaim. #224's fence makes that safe at the row level — it is not cancellation, and shouldn't be read as such. Threading an AbortSignal through JobHandlerContext is the actual fix.
Handlers with external side effects need idempotency keys
A database fence can't un-send an email. In an interleaving where a live claim is reclaimed, the first execution's external effects have already happened. lockUntil (#224) makes that much rarer, not impossible.
AI_RESPONSE— covered by #191'sMessage.ticketId_responseKeyand theresponseStatemachine.ESCALATION,HUBSPOT_SYNC,TRACKER_SYNC— not covered, and #224 is what introduces automatic re-execution.
Smaller
void currentPoll.finally(cb)builds a derived promise with no rejection handler.poll()can't reject today, so latent — but if a throw ever escapes it's an unhandled rejection, andstop()'sawait this.pollPromisewould propagate out so$disconnect()never runs.job-cleanup.tsnever deletesFAILEDrows at all, which is what makes the accumulation above unbounded.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing poll(), schedulePoll(), buildHealthResponse, claimAndProcessJobs, and job-cleanup.ts, then inspect runWithTimeout, JobHandlerContext, and stop(). Define the changes as separate concerns for polling, health, failed-job handling, shutdown, and external side effects. Done means concurrency accounting and health remain accurate while failure, timeout, and shutdown behavior are safe and bounded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100