CopilotKit / CopilotKit/outpost

worker: stop awaiting each claim batch, so concurrency limits and /health mean something

Open
#232 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: infrastructure bug roadmap roadmap: next
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

From the review of #224. These are the items that PR deliberately left out, and the first one is load-bearing for two others.

The batch await makes concurrency accounting dead code

poll() fully awaits Promise.allSettled over each claimed batch, and schedulePoll only runs after the poll resolves. So activeJobs.size and activeJobsByType are always 0 when availableSlots and typeLimit - activeForType are computed. Both expressions are unreachable: mutating each to ignore in-flight work passes the whole suite.

It does not over-fetch — the serialization accidentally bounds it — but per-type concurrency limits never bind, which is not what concurrencyByType looks like it does.

Which is also why /health is still wrong (#182, #184)

Because poll() blocks for the whole batch duration, lastPollTime stalls for as long as the longest job. buildHealthResponse compares it against STALE_POLL_MS = 60_000 with no active-job exemption, against a 120s AI_RESPONSE timeout. So a healthy worker mid-job returns 503 "stalled" and Railway restarts it — killing the job it was in the middle of.

Two ways out, and they should be decided together:

  1. Stop awaiting the batch (fixes the accounting too), or
  2. Exempt active jobs from the stale-poll check.

(1) is the better fix and makes (2) unnecessary. This is the reason /health has now been deferred three times; it should be one PR with the batch change.

An unregistered job type is terminally destroyed

FAILED with completedAt and no attempt increment — and job-cleanup.ts only deletes COMPLETED and DEAD_LETTER, so the rows accumulate forever. #224 added lockedAt: null, claimToken: null to that write, which means reclaim can't rescue them either.

Reachable only via claimAndProcessJobs, and production sets concurrencyByType so it takes the type-filtered path. Latent today; live the moment concurrencyByType is emptied or a worker ships without the full handler set.

A no-op release back to PENDING with no attempt consumed is the safe failure. Data-destroying path, so it wants its own change.

The timeout doesn't cancel the handler, and the drain has no deadline

runWithTimeout stops waiting for the handler; the handler keeps running. And AI_RESPONSE at 120s against Railway's ~30s SIGTERM grace means the graceful drain resolves well after SIGKILL already landed.

So the real shutdown path is still crash-abandonment plus reclaim. #224's fence makes that safe at the row level — it is not cancellation, and shouldn't be read as such. Threading an AbortSignal through JobHandlerContext is the actual fix.

Handlers with external side effects need idempotency keys

A database fence can't un-send an email. In an interleaving where a live claim is reclaimed, the first execution's external effects have already happened. lockUntil (#224) makes that much rarer, not impossible.

  • AI_RESPONSE — covered by #191's Message.ticketId_responseKey and the responseState machine.
  • ESCALATION, HUBSPOT_SYNC, TRACKER_SYNCnot covered, and #224 is what introduces automatic re-execution.

Smaller

  • void currentPoll.finally(cb) builds a derived promise with no rejection handler. poll() can't reject today, so latent — but if a throw ever escapes it's an unhandled rejection, and stop()'s await this.pollPromise would propagate out so $disconnect() never runs.
  • job-cleanup.ts never deletes FAILED rows at all, which is what makes the accumulation above unbounded.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing poll(), schedulePoll(), buildHealthResponse, claimAndProcessJobs, and job-cleanup.ts, then inspect runWithTimeout, JobHandlerContext, and stop(). Define the changes as separate concerns for polling, health, failed-job handling, shutdown, and external side effects. Done means concurrency accounting and health remain accurate while failure, timeout, and shutdown behavior are safe and bounded.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.