monad-developers / monad-developers/ultrafuzz

stats counts operator-cancelled nodes as failed while status counts them as other

Open
#1,087 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Summary

ultrafuzz status and ultrafuzz stats disagree about the same run. A node that was interrupted by an operator-requested ultrafuzz cancel is counted as other by status and as failed by stats.

Asking "did anything fail?" gives a different answer depending on which command you ask.

Observed

One run, cancelled on a timer while two nodes were mid-flight:

$ ultrafuzz status --project . linux-run2
Progress: 3% (8 finished / 0 running / 212 pending / 0 failed / 2 other / 222 total)

$ ultrafuzz status --project . linux-run2 --json
{"percent": 3, "finished": 8, "in_progress": 0, "pending": 212,
 "failed": 0, "skipped": 0, "remaining": 214, "total": 222}

$ ultrafuzz stats linux-run2 --project .
  12 succeeded
   2 failed        <-- threat-model, setup-foundry
  77 pending

status is right. Both nodes were interrupted by the cancel, not by any failure of their own:

error: "CLI aborted See https://smithers.sh/reference/errors"
warn  child process interrupted process:/usr/local/bin/bun runId=ultrafuzz-linux-run2

Neither node had failed on its own terms; each was executing when cancellation arrived.

Why it is worth fixing

A cancelled campaign is a normal outcome -- a time-boxed run, a budget stop, an operator changing their mind. Reporting the nodes that happened to be executing at that moment as failed invites a hunt for a defect that does not exist. I went looking for two bugs before finding CLI aborted in the log and realising the cancel had caused both.

It also makes the two views hard to reconcile: status says nothing failed, stats names two failures, and nothing in either output explains the difference.

stats additionally lists 91 rows against a 222-node graph -- presumably only nodes with recorded attempts -- which is reasonable on its own, but combined with the conflicting failed count it makes the two commands look like they are describing different runs.

Expected

A node interrupted by cancellation should be reported distinctly from one that failed -- cancelled, aborted, or interrupted -- in stats as it already is in status. Failing that, the two commands should at least agree on the count.

Environment

Host — macOS 26.6.2 (Darwin 25.6.0) arm64; podman 6.0.2 client, applehv machine, 4 CPU / 8 GiB / 60 GB

Podman VM — podman server 6.0.2 linux/arm64; kernel 7.1.3-200.fc44.aarch64 (Fedora CoreOS 44); crun 1.28

Containerdocker.io/library/node:24-bookworm (sha256:78b162211207872503ea9245188122b815150b9b4380e47a7c4a447332c01660), Debian GNU/Linux 12 (bookworm), aarch64, running as uid 1000, --memory 7g --cpus 4

Toolchain — node v24.20.0, pnpm 11.1.1, bun 1.4.0, forge 1.8.1, git 2.39.5, claude 2.1.258 (subscription, authMethod: claude.ai)

Ultrafuzzmain (8b985b90) plus #1083, built in-image

Targetmorpho-org/metamorpho-v1.1 @ 89de526, default audit profile, claude-opus-5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how ultrafuzz status and ultrafuzz stats classify node outcomes, using the reproduced cancellation commands and output as the behavioral baseline. Verify the handling of CLI aborted and interrupted child processes; done means cancelled nodes are represented distinctly or both commands report the same count, with coverage for an operator-cancelled run.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.