CopilotKit / CopilotKit/outpost

worker: /health reports ok for a stopped or wedged worker (revisits #138)

Open
#182 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: infrastructure roadmap roadmap: now
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

Found during the review of #180; the fix attempted there was reverted because it broke the healthy path (see below).

apps/worker/src/index.ts returns 200 {status:'ok', ...health} whenever the worker object exists. It never consults running or lastPollTime, so:

  • Stopped worker reports healthy. Worker.stop() sets running = false without exiting the process — including from Worker's own signal handlers — leaving /health serving 200 {"status":"ok","running":false}.
  • Wedged poll loop reports healthy. Worker.poll() catches every error and reschedules. A poll blocked on a hung database call leaves running: true and lastPollTime frozen, and /health returns 200 forever while the queue backs up.

Important constraint for whoever picks this up: the naive fix (503 when lastPollTime is stale) breaks the healthy path. Worker.poll() stamps lastPollTime at the top and then awaits the job batch, so the timestamp legitimately freezes for the whole batch — up to the 300s jobTimeouts. A 60s staleness bound therefore 503s a perfectly healthy worker mid-job and lets the container healthcheck kill it. That regression was caught in review and is why this is not already fixed.

Any real fix probably needs Worker to stamp when it schedules the next poll rather than when one begins, or to gate the stalled verdict on activeJobCount === 0.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read apps/worker/src/index.ts and inspect Worker.stop() and Worker.poll(), especially how running and lastPollTime are updated around polling and job batches. Exercise /health with a stopped worker, a wedged poll, and a healthy long-running job; done means unhealthy states are rejected without causing a healthy in-progress batch to fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.