theam / theam/facility

A scheduled agent replays every occurrence it missed while the worker was down, one paid run per tick

Open
#330 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
71
Forks
64
Avg merge
15h 38m
Merged PRs (30d)
66

Description

What happens

AgentScheduler.claim advances a schedule by exactly one cron step, computed from the schedule's own stored nextRunAt rather than from now:

// services/api/src/agents/scheduler.ts
private async claim(schedule: typeof agentSchedules.$inferSelect, now: Date) {
  const nextRunAt = nextOccurrence(schedule.cron, schedule.timezone, schedule.nextRunAt);
  ...
}

tick() selects every schedule with nextRunAt <= now and claims one occurrence each, and the worker runs agent.schedules on * * * * *:

await boss.schedule("agent.schedules", "* * * * *", {});

So a schedule that fell behind catches up one occurrence per minute, and each caught-up occurrence starts a real story turn.

Why it matters

The backlog is paid work, not bookkeeping. stories.start provisions a workspace and dispatches an engine, so every replayed occurrence is a provider call charged to the project's monthly budget.

An hourly schedule after a day of worker downtime fires 24 turns, one per minute — a burst against a budget sized for one run an hour, in a window narrow enough that the preflight check sees the earlier ones as still in flight. A five-minute schedule after the same outage is 288. Nothing bounds it: the loop replays as many stale occurrences as the outage contains.

Downtime is not exotic here. pnpm dev restarts the worker whenever tsx watch sees a file change, and #35 describes the same class of interruption on a deployed instance.

The occurrences being replayed are also, by definition, stale. A security-audit at 05:00 Monday that runs at 14:32 Tuesday is not the audit anyone asked for; it is the same audit against a different repository state, and the run it would have preceded already happened.

Reproduction

No live provider needed — nextOccurrence is exported and pure:

import { nextOccurrence } from "./services/api/src/agents/scheduler.js";

// A schedule last due at 06:00, a worker that returns 24 hours later.
let nextRunAt = new Date("2026-09-06T06:00:00Z");
const now = new Date("2026-09-07T06:00:00Z");
let fired = 0;
while (nextRunAt <= now) {
  nextRunAt = nextOccurrence("0 * * * *", "UTC", nextRunAt); // what claim() does
  fired += 1;
}
console.log(fired); // 24 — one dispatch per tick until the backlog drains

What is already right

Worth saying, because the fix should not disturb it: the claim is a proper compare-and-swap on (nextRunAt, lastScheduledAt), so two workers cannot both claim the same occurrence, and syncProject takes a pg_advisory_xact_lock before rewriting a project's schedules. The concurrency story is sound. The gap is only in which occurrence is claimed after a gap.

Possible shapes

I have no strong preference between these and would rather agree on the semantics than send a patch that picks one silently.

Coalesce. Compute the next occurrence from now instead of from the stored nextRunAt, keeping the CAS on the observed value. A backlog of any size collapses to one run. Three lines, and it makes the loop level-triggered: it converges on "this schedule is due" instead of replaying every edge it missed.

Bound it. Keep catch-up but skip occurrences older than a deadline, the way a Kubernetes CronJob uses startingDeadlineSeconds and treats more than 100 missed schedules as an error worth surfacing rather than absorbing.

Make it the manifest's choice. A trigger declares whether a missed occurrence is worth running late. More faithful, and more surface than the problem probably deserves today.

Coalescing is what I would pick: for every agent kickstart ships — security-audit, ci-doctor, pr-reviewer — a missed run wants to happen once, now, not N times against a repository that has moved on. But "run every missed occurrence" is a legitimate contract for some schedules, and if that is the intent then the bug is that nothing bounds or reports the backlog.

Either way the loop should be able to say it fell behind. tick() returns { projects, due, scheduled, failures }; due counts rows, not how stale they were, so a 24-occurrence catch-up and a normal minute are indistinguishable in the log.

Scope

services/api/src/agents/scheduler.ts only. The file is untouched by the 0.12 rebuild and by every open pull request, and services/api/test/agent-scheduler.test.ts already exercises nextOccurrence with a fixed clock, so whichever semantics you choose is deterministically testable without new harness.

Happy to implement it once the intended behaviour is settled.

Claude Code helped

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with services/api/src/agents/scheduler.ts, especially claim(), tick(), and nextOccurrence(), then run services/api/test/agent-scheduler.test.ts with its fixed clock. First settle whether missed occurrences should be coalesced, bounded, or configurable; done means the chosen semantics are implemented in that file, deterministically tested, and stale-backlog behavior is observable.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.