monad-developers / monad-developers/ultrafuzz

No supported way to re-seal workflow control evidence: one divergent file permanently disables product-level status and reporting

Open
#674 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Summary

There is no supported way to re-seal or re-bless workflow control evidence. Once a run's sealed control files diverge for any reason, WORKFLOW_CONTROL_EVIDENCE_INVALID is permanent: ultrafuzz status refuses the run for the rest of its life, so no state sync, no completion gate, and no final report can ever be produced — even while the underlying Smithers run continues to execute and finish work.

Evidence

verifyWorkflowControlSnapshot compares the published snapshot's control files byte-for-byte (packages/runtime/src/workflow-integrity.ts:291):

throw new Error(`sealed workflow control file changed: ${controlFileLabel(key)}`);

which start-run.ts converts into the terminal diagnostic:

code:    WORKFLOW_CONTROL_EVIDENCE_INVALID
message: sealed workflow control file changed: workflow

In an 18-hour local run this fired from a single changed file. Verifying the seal by hand:

  • all 44,345 sealed execution_files hashed clean — zero mismatches
  • the only divergence was the published snapshot's .smithers/workflows/<run>.tsx:
seal expects    f37c4eb6…  2007734 bytes
generated copy  f37c4eb6…  2007734 bytes   ← still byte-identical to the seal
snapshot copy   96c367e8…  2008029 bytes   ← diverged

So the damage was one file out of 44,346, and the pristine bytes were still sitting on disk. There is still no way to recover, because sealWorkflowControlFiles refuses to write a second seal (workflow-integrity.ts:233):

if (pathEntryExists(paths.integrityPath)) {
  throw new Error("workflow control seal already exists");
}

There is no --reseal, no --accept-control-change, no generation bump. The run is bricked at the product level regardless of how repairable the underlying divergence is.

Why the divergence happened

Because agentic nodes compile with retries: 0 (filed separately), the only available way to grant a failed node another attempt was to patch the sealed snapshot's workflow. Smithers itself accepts this via smithers up --force --accept-workflow-change, and the run genuinely resumed and kept finishing nodes. Ultrafuzz then had no matching concept, so the product layer was locked out of a run that was still alive and progressing.

That asymmetry is the core defect: Smithers has a supported "I acknowledge the workflow changed" path and Ultrafuzz does not.

Impact

  • One divergent file permanently forfeits product-level status, sync, completion gating, and final reporting for that run.
  • The failure is silent in the wrong direction: smithers status reports a healthy running run while ultrafuzz status refuses it, so a run can look fine and be unfinishable.
  • Long runs are the ones most likely to need mid-flight recovery and the most expensive to discard, so the policy bites hardest exactly where it costs most.

Proposal

Add a supported re-seal path that preserves auditability rather than discarding it:

  • ultrafuzz reseal <run-id> (or --accept-control-change on the existing resume path) that recomputes the seal, bumps the control generation, and appends a provenance record: what changed, old and new digests, timestamp, and operator-supplied reason.
  • Keep the strict default. Re-sealing should be explicit, logged, and visible in the final report as a deviation, never implicit.
  • Report and validation output should surface "this run's control evidence was re-sealed N times" so a re-sealed run is never mistaken for a pristine one.
  • Consider a --dry-run that reports exactly which control files diverged and whether pristine bytes are still recoverable, which would have reduced the diagnosis here from hours to seconds.

Acceptance criteria

  • A run whose control evidence diverged can be returned to a verifiable state without discarding completed work.
  • Re-sealing bumps the generation and records an auditable provenance entry.
  • Re-sealed runs are explicitly marked as such in status and report output.
  • A diagnostic names the specific divergent control files instead of only the first one it hits.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with verifyWorkflowControlSnapshot and sealWorkflowControlFiles in packages/runtime/src/workflow-integrity.ts, then trace how start-run.ts turns verification failures into WORKFLOW_CONTROL_EVIDENCE_INVALID. Compare the existing ultrafuzz status and resume paths with the stated reseal proposal. Done means divergent control evidence can be explicitly re-sealed with generation, provenance, status, report, and diagnostic updates without discarding completed work.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, cli, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.