monad-developers / monad-developers/ultrafuzz

Terminal workflow stays running in status after unrecorded attempt supersession

Open
#1,099 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

A recovered workflow can finish every remaining task and publish a verified final report while ultrafuzz status --json still returns ok: false, status: running, workflow_status: finished, zero remaining tasks, and ETA zero.

The observed diagnostic was:

WORKFLOW_ATTEMPT_INSPECT_FAILED
Smithers attempt ["node:synthetic-worker",0,1] supersedes terminal event <sequence> before durable attempt recording

RUN_WORKFLOW_STATUS_DIVERGED
Ultrafuzz run is running while the workflow runner reports terminal finished

The node name and event sequence above are placeholders. The workflow had undergone recovery that reused attempt numbers; the final producer and verifier were complete. This is a historical bookkeeping failure, not unfinished report generation. The report and native bundle were independently readable and verified.

On current main, terminalWorkflowAttempts in packages/runtime/src/workflow-sync.ts rejects a superseded terminal occurrence unless it was already recorded or an explicit authority callback admits it. In the observed recovery history, that exception prevents terminal run-state reconciliation. The exact missing authority needs an isolated reproducer; this report does not propose accepting arbitrary supersession.

Suggested regression fixture:

  1. Use a model-free workflow that finishes an attempt.
  2. Retain that event history, then reopen the same node/iteration/attempt identity through a supported recovery boundary before its old terminal occurrence is projected.
  3. Finish the replacement and final report, then synchronize status and stats twice.

Expected: retain each authenticated historical occurrence under a distinct durable identity and report terminal completion consistently. If part of the history remains unrecoverable, expose a bounded history diagnostic without presenting the already-finished runner as actively executing. Observation must not reset tasks or rewrite historical results.

Related: #951, #1036, #1077. This issue tracks the terminal-reconciliation impact and the unrecorded-supersession diagnostic, rather than the different immutable-ledger collision in #1036.

Sanitized: no target names, findings, run identifiers, host paths, credentials, or raw campaign artifacts are included.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/runtime/src/workflow-sync.ts at terminalWorkflowAttempts and trace the status and stats synchronization path. Build the suggested model-free recovery history with a reused attempt identity, then synchronize twice. Done means authenticated historical occurrences retain distinct durable identities, terminal completion is reported consistently, and unrecoverable history produces a bounded diagnostic without resetting tasks or rewriting results.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.