monad-developers / monad-developers/ultrafuzz

Add an enforceable recovery plan that protects already-finished tasks

Open
#1,102 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

A recovery command can reopen a large number of already-verified tasks when the operator intended to recover one failed preparation or verifier. In the observed incident, the explicit downstream reset reopened completed work, and dynamic-generation recovery also reopened finished tasks outside the previously calculated closure. Durable artifacts often survived, but the scheduler scheduled expensive model work again.

Local operational safeguards now prevent this for that run. There is still no product-level guarantee that a retry will preserve completed work or stop before exceeding its advertised impact. --keep-workspaces alone does not provide that guarantee: preserving files and preserving successful scheduler state are different operations.

Related existing issues cover result reuse (#1077), reuse when prompts change (#1079), stale preparation authority (#1081), and progress counters (#970). This issue concerns pre-mutation recovery planning and enforcement.

Suggested synthetic fixture: build a graph with one failing preparation, several finished dependents, and a separate finished dynamic group. Request a reset or failed-node retry. Compute the complete affected set, including runtime dynamic-generation withdrawal, before changing the database or artifact locations. Assert that execution cannot reopen any task outside that set.

Acceptance criteria:

  • Expose a machine-readable recovery plan listing affected preparation, producer, and verifier tasks, including currently finished tasks and any dynamic-generation changes.
  • Make the plan available without performing a reset.
  • Offer a progress-preserving mode that refuses to reopen a finished task; require an explicit override when an operator intentionally requests that loss of current progress.
  • Revalidate the planned set at execution time and stop before mutation if it has changed.
  • Test that refusal preserves scheduler rows, verified artifacts, prompts, and dependency authority.
  • Provide a documented verifier-only/final-report recovery path whose affected set excludes upstream successful work.

The observed beyond-closure reopening involved local recovery patches; a current-main integration fixture is still required to determine the exact current exposure. This issue must not be read as a claim that all reset paths on main exhibit that same collateral behavior.

Sanitized: all task descriptions are generic; no target names, finding content, private logs, run identifiers, or host paths are included.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the current-main recovery and reset entry points, including where scheduler rows, database state, and artifact locations are mutated; the issue names no specific files or tests. Build the suggested synthetic graph and integration fixture, then verify that planning, refusal, revalidation, and verifier-only recovery preserve successful work and limit changes to the affected set.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, database, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.