monad-developers / monad-developers/ultrafuzz
workflow_deadline_seconds is only enforced while an operator runs a CLI command; unattended runs have no wall clock
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Summary
[run].workflow_deadline_seconds is only enforced while an operator is running a CLI command. The deadline check lives inside synchronizeLinkedWorkflowRun, and nothing calls that function on a timer -- so an unattended campaign has no enforced wall clock at all. It runs until someone types ultrafuzz status.
A run configured with a one-hour deadline ran past it and kept going. The deadline was computed correctly and simply never evaluated.
This is not agent-specific; see "Not agent-specific" below.
Observed
Campaign mm-1h-0905-2205, MetaMorpho v1.1, 222-node graph, launched with workflow_deadline_seconds = 3600.
The deadline was set correctly:
started: 2026-09-05T22:06:55.396Z
deadline: 2026-09-05T23:05:06.091Z
At 23:08 -- three minutes past -- the run was still executing, with live agent processes and artifacts being written (setup-foundry/setup/setup-foundry.md at 23:07, base-test-setup/workspace-patch-preparation.json at 23:08). No cancellation, and no workflow-deadline-exceeded event.
Two further symptoms with the same cause:
state.json never advanced. last_transition_at was byte-identical to started_at for the whole hour, and the embedded concurrency probe was stamped before the workflow was even submitted:
last transition: 2026-09-05T22:06:55.396Z (== started_at)
concurrency.observed_at: 2026-09-05T22:05:06.091Z
workflow-submitted event: 2026-09-05T22:06:55.408Z
node counts, whole run: {'succeeded': 10, 'pending': 81}
Those 10 "succeeded" are the reference nodes, materialized from cache before submission. No agentic node was ever recorded.
events.jsonl stopped at 15 lines, the last being workflow-submitted.
Both were wrong. The Smithers controller log for the same run shows five nodes actually executing:
[00:00:05] -> node:project-discovery
[00:24:23] -> node:actors-flows
[00:38:25] -> node:threat-model
[00:38:25] -> node:setup-foundry
[01:00:41] -> node:base-test-setup
and four of them wrote complete primary artifacts (project-discovery.md 127,201 B; invariant-evidence-ledger.json 95,771 B; actors-flows.md 58,061 B; setup-foundry.md 13,450 B).
So the engine knew the truth throughout. The project-local ledger was frozen at plan time, and the deadline was never consulted.
Mechanism
The deadline is evaluated at packages/runtime/src/workflow-sync.ts:1305-1345:
if (workflowControl.deadlineExceeded) { ... cancel ... emit "workflow-deadline-exceeded" }
That block is inside synchronizeLinkedWorkflowRun. Every call site is operator-initiated:
| call site | reached by |
|---|---|
state-export.ts:209 (in getRunHealth) |
ultrafuzz status |
state-export.ts:119 (in getRunStatus) |
ultrafuzz inspect |
lifecycle-inspection.ts:290 (in diagnoseRun) |
ultrafuzz why |
cli/src/commands/stats.ts:138 |
ultrafuzz stats |
workflow-sync.ts:974 (syncRun wrapper) |
start-run.ts:751, once at launch |
There is no timer: grep -rn 'setInterval' packages/runtime/src packages/cli/src returns nothing. ultrafuzz run returns as soon as the workflow is submitted, having synced exactly once, and no process synchronizes again.
Note that cancelRun (lifecycle-inspection.ts:229) does not synchronize. That matches observed behaviour: ultrafuzz cancel reported "is stopping" and terminated the agent processes, but state.json still read running afterwards.
Why it matters
A time box that only fires when watched is not a time box. The failure mode is silent and expensive: an operator sets workflow_deadline_seconds, walks away, and the campaign bills indefinitely.
There is also an unfortunate interaction. The only way to enforce the deadline is to run a synchronizing command, and ultrafuzz status is itself capable of hanging -- in this session it exceeded a 120-second timeout on a live run and had to be killed. So the enforcement path is gated behind a command that is not reliably available.
Finally, because the ledger is only refreshed by those same commands, state.json and events.jsonl cannot be used to answer "is this run progressing?" A stalled run and a healthy unattended run are byte-identical on disk. That ambiguity is what made an earlier healthy run look dead; it was terminated on that basis.
Not agent-specific
Worth stating, because it is the natural first guess:
- The deadline block at
workflow-sync.ts:1300-1335contains no agent, model, engine, or provider conditional. workflow-sync.tsis 5,891 lines and references an agent identifier exactly twice, at:4685and:5571, both writingagent_refinto an output record. Data, not control flow.- None of the 11 adapter templates under
packages/runtime/src/templates/smithers/agents/referencesynchronizeLinkedWorkflowRunorworkflow_deadline.
The only agent-scoped deadline in the tree is OPENROUTER_ATTEMPT_DEADLINES (openrouter.tsx:31), a per-attempt provider retry budget, unrelated to the workflow wall clock.
Every backend is affected.
Possible resolutions
- Enforce the deadline where the work runs. The Smithers controller is alive for the whole campaign and already knows the wall clock; handing it the deadline at submission makes enforcement independent of any operator.
- Failing that, sync on a timer. A lightweight periodic
synchronizeLinkedWorkflowRunfor live runs would fix the deadline,state.jsonstaleness, andevents.jsonlstaleness together. - Make
cancelRunsynchronize, so a cancellation is reflected in the ledger rather than leaving a terminated run readingrunning. - At minimum, document the constraint and warn at launch when
workflow_deadline_secondsis set, since today the setting reads as a guarantee it does not provide.
Items 1 and 2 are alternatives. 3 and 4 stand on their own.
Environment
- image
localhost/ultrafuzz:main2, buildf9a3646f78badca09525e99586ca071da489f4e6 @ultrafuzz/cli/0.1.0 linux-arm64 node-v24.20.0- run
mm-1h-0905-2205, MetaMorpho v1.1, 222-node graph,ClaudeAgent
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/runtime/src/workflow-sync.ts around synchronizeLinkedWorkflowRun and its call sites in state-export.ts, lifecycle-inspection.ts, cli/src/commands/stats.ts, and start-run.ts. Trace how the Smithers controller and live runs are managed before choosing between controller-side enforcement and periodic synchronization. Done means an unattended run enforces workflow_deadline_seconds and the resulting cancellation and run state are reflected without an operator command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100