CopilotKit / CopilotKit/outpost

worker: the scheduler double-emits every recurring job if outpost-worker scales past 1

Open
#198 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: infrastructure roadmap roadmap: next
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

packages/outpost/queue/src/scheduler.ts:79 schedules recurring jobs with a plain in-process interval:

const timer = setInterval(() => this.tick(definition), definition.intervalMs);

apps/worker/src/index.ts:94 constructs one Scheduler per process and starts it at :117. With a single outpost-worker replica this is correct. At two replicas, both tick, and every scheduled job type — SLA_CHECK, ONBOARDING_DIGEST, GITHUB_REACTION_POLL, ACCOUNT_SCORING, JOB_CLEANUP — is enqueued twice per interval.

Why it matters

It is a scaling landmine rather than a live defect: nothing warns you, and the first symptom would be duplicate outbound posts from the digest, which is exactly the class of bug #133 was filed to prevent.

Fix

Leader election on the SystemConfig table — a lease row the scheduler must hold, renewed each tick, so exactly one replica emits. Alternatively, split the scheduler into its own single-replica service. Either way, worth landing before anyone scales the worker, not after.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/outpost/queue/src/scheduler.ts:79 and apps/worker/src/index.ts:94,117 to trace scheduler construction and startup. Inspect the SystemConfig table and existing issue #133 for relevant coordination context. Done means recurring jobs are emitted by only one worker replica when outpost-worker scales past one, without duplicate enqueues.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.