Run each node on the cloud

Summary

Add a provider-neutral execution layer that allows a regular ultrafuzz run to execute each concrete expanded agentic node attempt on its own cloud VM. Modal is the first provider; the design must leave room for later providers such as ECS.

Local execution remains the default. Users opt into cloud execution and configure global resources, provider settings, credentials by environment-variable name, retention, and per-node overrides in ultrafuzz.toml.

Motivation

Today, an Ultrafuzz workflow executes its nodes within one worker environment. This prevents a single run from:

• assigning a full multi-core machine to each heavy fuzzing or agentic node;
• scaling independent DAG branches across separate machines;
• using different machine sizes for different nodes; and
• replacing a worker at a node boundary when a cloud provider imposes a per-machine lifetime limit (Modal currently limits a sandbox to 24 hours).

The desired model is a local Ultrafuzz controller coordinating durable node executions in the cloud. Each completed node publishes enough durable state for its dependents and for a later ultrafuzz resume invocation.

Product contract

Execution unit

• One concrete expanded agentic attempt runs in one fresh cloud VM/sandbox. Expansion includes topology loops and model fan-out.
• The attempt's agent execution and output validation run in the same VM.
• A retry or replacement is a new VM, not reuse of a failed VM.
• Meta and reference nodes remain local unless a later design explicitly gives them remote work.
• Independent ready nodes may run concurrently, subject to existing Ultrafuzz concurrency limits and provider capacity.
• A successful VM must not be reused to execute a second concrete agentic attempt.

User-facing configuration

Extend ultrafuzz.toml with a provider-neutral runner/execution section. The exact key names may follow existing configuration conventions, but the schema must support all of the following:

• mode/type: local or cloud, defaulting to local;
• cloud provider selection, initially modal;
• provider credentials by environment-variable name, never inline secrets;
• provider-specific settings such as app/image and region when supported;
• global defaults for CPU, memory, timeout, and other provider-supported resources;
• overrides keyed by logical topology node ID;
• persistent-volume retention/TTL, defaulting to 30 days and overrideable per project;
• validation that rejects cloud mode without a provider, unknown providers, invalid node IDs, invalid resource values, missing credential environment variables at execution time, and provider settings placed under the wrong provider.

The provider boundary must not bake Modal concepts into topology expansion or the core scheduler. Modal-specific launch, status, cancellation, storage, and cleanup behavior belongs behind an execution-provider interface so another provider such as ECS can be added without changing topology semantics.

config.resolved.toml and other run evidence must record the selected mode, provider, resolved non-secret resource settings, and node overrides. Secrets and credential values must never be persisted.

Backward compatibility

• Existing ultrafuzz.toml files continue to validate and run locally without modification.
• Local mode remains the default and preserves current topology, retry, concurrency, resume, replay, and fork behavior.
• Cloud mode uses the same planned graph, prompt rendering, artifact contracts, attempt ledger, reporting, and materialization behavior as local mode.

Durable handoff design decision

The precise workspace handoff contract is intentionally left to the implementer, but it must be decided and documented before implementation. The design must state whether a dependent receives:

1. a complete immutable snapshot of a parent's mutated worktree;
2. a clean checkout plus declared artifacts/messages; or
3. a hybrid of those approaches.

The design must also define fan-in behavior for nodes with multiple parents. It must not silently select one parent's workspace or perform an implicit lossy merge.

Whichever approach is selected must satisfy these constraints:

• dependency inputs are immutable and content-addressed or otherwise identity-checked;
• a child sees only successfully published dependency state;
• parallel children cannot mutate shared parent state;
• publication is atomic, so partial workspaces or artifacts are never treated as complete;
• paths, symlinks, and archive extraction are validated using the same safety posture as existing Ultrafuzz artifacts;
• credentials, auth files, provider secrets, and unrelated host files are excluded;
• retries receive the same dependency inputs as the original attempt;
• the selected single-parent and multi-parent semantics are covered by integration tests and documented for users.

Use Smithers workflow abstractions where they provide the remote task, durable state, checkpoint, or continuation primitives; keep Smithers implementation details behind Ultrafuzz's existing workflow boundary.

Controller and resume behavior

The main Ultrafuzz controller starts on the developer machine.

For the initial implementation, the controller does not need to schedule new nodes while the developer machine is offline. However:

• already-launched cloud nodes may finish and atomically publish their results after the local process exits;
• rerunning ultrafuzz resume <run-id> must reattach to the same cloud run, reconcile provider state, reuse successfully completed attempts, and schedule remaining nodes;
• resume must not launch a duplicate VM for an attempt that is running or whose successful publication can be proven;
• controller loss during launch must be handled with durable launch ownership/idempotency records rather than optimistic duplicate submission.

This behavior must integrate with the existing controller lease, workflow evidence, attempt ledger, and reset-node behavior.

Failure, timeout, and cancellation semantics

• Node output is committed atomically only after agent execution and output validation succeed.
• A VM failure, provider interruption, or lifetime limit leaves the attempt incomplete and eligible for the existing retry policy.
• A retry/replacement VM starts from the last successfully published dependency state and reruns only the incomplete node.
• Completed dependencies and completed sibling nodes are not repeated.
• Cancellation stops or requests termination of all owned cloud VMs and records any provider failures without losing already-published evidence.
• resume, replay, fork, and --reset-node must have documented cloud semantics and must not violate attempt or workspace lineage.
• In-node checkpoint continuation is optional for this issue; node-boundary continuation and replacement are required.

Storage lifecycle

• Use private, run-scoped durable cloud storage. For Modal, use a private Modal Volume or an equivalent provider primitive.
• Store only the run state, dependency handoffs, workspaces required by the selected handoff design, logs/evidence, and artifacts necessary to resume or inspect the run.
• Flush/publish state before declaring a node successful.
• Reconcile final artifacts and run evidence back to the local project so existing inspect, report, and materialization commands continue to work.
• Default retention is 30 days, configurable in ultrafuzz.toml.
• Provide an explicit cleanup path and make cleanup idempotent. Cleanup must not remove an active run's storage without a clear force/confirmation path.
• Credentials remain ephemeral and must not be written to images, volumes, logs, artifacts, or resolved config.

Observability

Run evidence and ultrafuzz inspect must expose enough provider-neutral information to diagnose a distributed run:

• execution mode and provider;
• logical node ID, concrete attempt ID, and retry index;
• provider execution/VM ID and durable storage lineage;
• requested and resolved resources;
• queued, launching, running, publishing, succeeded, failed, cancelled, and provider-unknown states;
• timestamps, terminal reason, and whether an attempt was executed, resumed, or reused;
• sanitized provider errors and cleanup state.

Provider IDs and state must be persisted before a launch can be considered owned by the run.

Acceptance criteria

Automated coverage

• Unit tests cover configuration parsing/defaults, provider selection, per-node resource overrides, secret redaction, invalid configuration, and local-mode backward compatibility.
• Provider-contract tests cover idempotent launch, status reconciliation, cancellation, cleanup, atomic publication, and normalized provider errors.
• DAG integration tests cover a dependency chain, parallel fan-out, multi-parent fan-in under the selected handoff contract, retries, and reuse of completed attempts.
• A resume integration test terminates the local controller after at least one node is launched or completed, then proves ultrafuzz resume completes the same run without duplicating completed or live attempts.
• A forced short-timeout/interruption test proves that a replacement VM reruns only the incomplete node from the last successful dependency state.
• Existing local execution tests continue to pass.

Real Modal validation

Run the cloud implementation against all three checked-in Ultrafuzz-bench smoke targets:

• very-liquid-vaults-foundry
• venus-isolated-pools-hardhat
• stableswap-ng-vyper

For each target, retain evidence showing:

• every concrete agentic attempt ran in a distinct Modal sandbox/VM;
• global resource settings were applied and at least one node-specific override was honored;
• dependency handoffs followed the documented workspace/artifact contract;
• the run produced a valid final report whose artifacts are available through the normal local Ultrafuzz commands; and
• no completed node was repeated during a demonstrated stop-and-resume cycle.

The real-cloud test may be gated/manual to control cost, but its command/configuration and sanitized result evidence must be reproducible and documented.

Documentation and deliverables

• Document the provider interface and the chosen workspace/fan-in handoff contract.
• Add an ultrafuzz.toml example for local mode and Modal cloud mode, including per-node overrides and credential environment-variable names.
• Document authentication, resource/cost implications, status inspection, resume after controller loss, cancellation, retention, and cleanup.
• Update architecture documentation so cloud execution remains behind Ultrafuzz's workflow boundary.
• Include sanitized evidence from the three real Modal validation runs in the pull request or a linked artifact.

Start by tracing the existing workflow boundary, controller lease, attempt ledger, local execution, and resume behavior, then inspect the ultrafuzz.toml and config.resolved.toml flows. Define and document the provider interface and durable workspace/fan-in contract before implementing cloud execution; completion requires the listed unit, provider, DAG, resume, interruption, local-regression, and gated Modal smoke tests.