monad-developers / monad-developers/ultrafuzz
goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Summary
Follow-up to #645. That issue raised the default-profile per-agent timeout, but the goals and strategies groups pin timeout_seconds: 7200 in the shipped topology, and the group pin wins over the profile/config default. High-reasoning routed models are still killed at the 2h boundary, and a timeout kill discards the node's entire output.
Evidence
.ultrafuzz/topology.yml group defaults:
groups:
goals:
defaults:
loops: 1
timeout_seconds: 7200
model_profiles: [default]
strategies:
defaults:
loops: 3
timeout_seconds: 7200
model_profiles: [default]
The resolution order in packages/runtime/src/smithers.ts:3734 takes the node value first, so the group pin shadows both the profile and run.default_timeout_seconds:
const timeoutMs =
(input.node.timeoutSeconds ?? profile.timeoutSeconds ?? input.config.run.defaultTimeoutSeconds) * 1000;
In an 18-hour run configured with default_timeout_seconds = 3600, goal nodes were killed at exactly the group pin, not the config value:
+07:23:46 NodeStarted → node:<dynamic-goal-a> (attempt 1, iteration 0)
+09:23:50 NodeFailed ✗ node:<dynamic-goal-a> (attempt 1): CLI timed out after 7200000ms
+09:07:26 NodeStarted → node:<dynamic-goal-b> (attempt 2, iteration 0)
+11:07:32 NodeFailed ✗ node:<dynamic-goal-b> (attempt 2): CLI timed out after 7200000ms
Nine failures in the run were this exact 7200000ms kill. Every one lost the whole node: the artifact directories for timed-out goal nodes contain only prompt.rendered.md and workspace-patch-preparation.json, no published output.
Two distinct problems
The window is too short for the workload. Retrying does not help, because a node that needs more than 2h of high-reasoning wall time hits the identical wall on attempt 2. Observed: a node killed at 2h on attempt 1, restarted, and on track to be killed at 2h again.
A timeout is total loss. There is no finalisation reserve, so ~2h of model spend produces nothing. #645 listed "ensure the finalization reserve is large enough for artifact writing and validation after long reasoning phases" in its proposal, but the group pins were not part of that change, so goal and strategy nodes never got either half of it.
Proposal
- Raise
timeout_secondson thegoalsandstrategiesgroup defaults, or drop the pins so the profile/config value applies. - Make the timeout model-aware, as #645 proposed, and apply the multiplier to group defaults rather than only to the profile default.
- Add a finalisation reserve: signal the agent before the hard deadline so it can publish partial output, and treat a node that published a valid artifact before being cut off as a success rather than a total loss.
- Consider validating at plan time that no group pin is below the resolved profile timeout, and warn when it is — that would have surfaced this as a config warning instead of nine dead nodes.
Acceptance criteria
- Goal and strategy nodes running high-reasoning routed models complete within their window under the shipped
defaultprofile. - A group
timeout_secondsbelow the resolved profile timeout produces a plan-time warning. - A node cut off by the deadline retains any artifact it had already published.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .ultrafuzz/topology.yml and packages/runtime/src/smithers.ts:3734 to trace how node, profile, group, and config timeouts are resolved. Review #645 and the timeout finalisation behavior before choosing the scope. Done means the shipped goals and strategies topology meets the acceptance criteria, including plan-time warnings and retention of already-published artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100