monad-developers / monad-developers/ultrafuzz

goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work

Open
#675 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Summary

Follow-up to #645. That issue raised the default-profile per-agent timeout, but the goals and strategies groups pin timeout_seconds: 7200 in the shipped topology, and the group pin wins over the profile/config default. High-reasoning routed models are still killed at the 2h boundary, and a timeout kill discards the node's entire output.

Evidence

.ultrafuzz/topology.yml group defaults:

groups:
  goals:
    defaults:
      loops: 1
      timeout_seconds: 7200
      model_profiles: [default]
  strategies:
    defaults:
      loops: 3
      timeout_seconds: 7200
      model_profiles: [default]

The resolution order in packages/runtime/src/smithers.ts:3734 takes the node value first, so the group pin shadows both the profile and run.default_timeout_seconds:

const timeoutMs =
  (input.node.timeoutSeconds ?? profile.timeoutSeconds ?? input.config.run.defaultTimeoutSeconds) * 1000;

In an 18-hour run configured with default_timeout_seconds = 3600, goal nodes were killed at exactly the group pin, not the config value:

+07:23:46  NodeStarted  → node:<dynamic-goal-a> (attempt 1, iteration 0)
+09:23:50  NodeFailed   ✗ node:<dynamic-goal-a> (attempt 1): CLI timed out after 7200000ms
+09:07:26  NodeStarted  → node:<dynamic-goal-b> (attempt 2, iteration 0)
+11:07:32  NodeFailed   ✗ node:<dynamic-goal-b> (attempt 2): CLI timed out after 7200000ms

Nine failures in the run were this exact 7200000ms kill. Every one lost the whole node: the artifact directories for timed-out goal nodes contain only prompt.rendered.md and workspace-patch-preparation.json, no published output.

Two distinct problems

The window is too short for the workload. Retrying does not help, because a node that needs more than 2h of high-reasoning wall time hits the identical wall on attempt 2. Observed: a node killed at 2h on attempt 1, restarted, and on track to be killed at 2h again.

A timeout is total loss. There is no finalisation reserve, so ~2h of model spend produces nothing. #645 listed "ensure the finalization reserve is large enough for artifact writing and validation after long reasoning phases" in its proposal, but the group pins were not part of that change, so goal and strategy nodes never got either half of it.

Proposal

  • Raise timeout_seconds on the goals and strategies group defaults, or drop the pins so the profile/config value applies.
  • Make the timeout model-aware, as #645 proposed, and apply the multiplier to group defaults rather than only to the profile default.
  • Add a finalisation reserve: signal the agent before the hard deadline so it can publish partial output, and treat a node that published a valid artifact before being cut off as a success rather than a total loss.
  • Consider validating at plan time that no group pin is below the resolved profile timeout, and warn when it is — that would have surfaced this as a config warning instead of nine dead nodes.

Acceptance criteria

  • Goal and strategy nodes running high-reasoning routed models complete within their window under the shipped default profile.
  • A group timeout_seconds below the resolved profile timeout produces a plan-time warning.
  • A node cut off by the deadline retains any artifact it had already published.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .ultrafuzz/topology.yml and packages/runtime/src/smithers.ts:3734 to trace how node, profile, group, and config timeouts are resolved. Review #645 and the timeout finalisation behavior before choosing the scope. Done means the shipped goals and strategies topology meets the acceptance criteria, including plan-time warnings and retention of already-published artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.