Reconsider prepare/node/verify setup

Context

A large default-profile campaign was analyzed to understand whether the prepare: and verify: stages consume too much time relative to the producer node: stage.

Durations were measured from each NodeStarted event to its matching NodeFinished or NodeFailed event. The dataset contained 678 completed executions; 34 in-flight starts were excluded. Standard deviations below are population standard deviations. Aggregate hours are summed task/agent time, so concurrently executing tasks overlap in wall-clock time.

The audited project and all campaign findings are intentionally omitted.

Successful executions

All completed attempts

This includes failures and retries.

The all-attempt node median is depressed by 221 failed attempts that often terminated quickly, so the successful-execution table better represents normal producer work.

Time allocation

Across all completed attempts:

• Node: 94.07%
• Prepare: 5.82%
• Verify: 0.12%
• Combined prepare/verify overhead: 5.93%

Across the 30 fully successful prepare -> node -> verify triplets:

• Node: 96.75%
• Prepare: 2.93%
• Verify: 0.32%
• Combined overhead: 3.25%
• Median per-triplet overhead: 2.02%
• Average per-triplet overhead: 3.55%
• P90 per-triplet overhead: 9.54%
• Maximum observed per-triplet overhead: 13.78%

Interpretation

Verification is effectively negligible. Preparation is modest in aggregate, but it has a meaningful long tail: its median is under one minute while P90 is approximately 6m 45s and the observed maximum is approximately 8m 36s.

The current structure is not an obvious global bottleneck, but the preparation outliers and repeated work across retries make the setup worth reconsidering.

Questions / possible directions

• Can preparation results be safely reused or cached across retries and related fanout nodes?
• Can preparation be made incremental so unchanged workspace state avoids repeated setup?
• Are any preparation steps better performed once at the fanout or campaign level?
• Can stage-level timing and cache-hit metrics be exposed directly by Ultrafuzz?
• Should long-running preparation emit more granular progress and attribution so filesystem, dependency, snapshot, and workspace costs can be distinguished?
• Can regression tests or benchmarks establish an acceptable preparation-overhead budget?

No files, tests, or entry points are named. Start by locating the prepare, node, and verify stage implementations and their retry or fanout boundaries, then identify which proposed direction is selected. Done would need a defined outcome such as a regression benchmark, cache behavior, or stage-level timing metrics, but the issue does not choose one.