Concurrent agents race on OAuth token refresh; three immediate retries all re-race and kill the run with an opaque "Claude run failed"

Summary

With subscription auth, concurrent agents share one provider home and race each other refreshing the OAuth token. Three of four lost the race, ultrafuzz retried all three immediately, each retry re-raced, and stall detection killed the campaign at its first agent node.

The operator sees only Error: Claude run failed. The actual cause is written to a Claude Code session transcript that ultrafuzz never surfaces.

What the agent actually reported

From $CLAUDE_CONFIG_DIR/projects/<munged-worktree>/<session>.jsonl, the assistant turn of each failed attempt:

Failed to refresh OAuth token: another Claude Code process is refreshing it or
exited mid-refresh. This is usually transient; retry in a minute, and if it
persists close other Claude Code processes or sign in again

Identical in all three failing sessions:

275ea901  Failed to refresh OAuth token: another Claude Code process is refreshi...
354278ea  Failed to refresh OAuth token: another Claude Code process is refreshi...
d66cdb74  Failed to refresh OAuth token: another Claude Code process is refreshi...

(Two further sessions in the same directory return OK -- those are manual probes I ran afterwards to confirm the credential and the exact argv still work.)

What ultrafuzz surfaced instead

node:project-discovery#0 stalled: Node stopped making progress: 3 consecutive
attempts failed with an identical error. Last error: Error: Claude run failed

Progress: 0% (1 finished / 0 running / 220 pending / 1 failed / 222 total)

and in the run log:

WARN agent.generate=7968ms: agent invocation failed {
  agentEngine: "claude-code", agentModel: "claude-opus-5",
  hasMessages: false, promptBytes: 17927,
  error: "Claude run failed",
}

hasMessages: false is the only hint that the agent produced nothing, and nothing anywhere in ultrafuzz's own output names OAuth, tokens, or concurrency.

Why the retries could not help

max_parallel_agents: 4
same_agent_attempts:  3

Four agents start together against one provider home. The three that lose the refresh race fail in about 8 seconds each, and the retries fire immediately -- so each retry re-enters the same race. The agent's own message says "retry in a minute"; all three attempts completed inside 25 seconds.

Because the three attempts fail with an identical error, stall detection classifies the node as making no progress and terminates the run. A transient condition with an explicit recovery window becomes a terminal failure.

Not a credential problem

Verified after the failure, in the same container, with the same credential:

• claude auth status -> loggedIn: true, authMethod: claude.ai
• rate limits fine: five_hour utilisation 0.02, seven_day 0.01
• the exact argv from the failing invocation, replayed by hand, returns "result":"OK", is_error: false
• replayed from inside the same isolated git worktree: also OK
• replayed with --add-dir pointing at the same artifacts directory: also OK

Every ingredient works in isolation. Only concurrency reproduces it.

Where the boundary lies

Claude run failed does not appear anywhere in this repository -- it comes from the vendored smthrs ClaudeCodeAgent. So the wording is not ultrafuzz's to change directly, but two things about the handling are:

1. The cause is discoverable and discarded. The explanation is in the session transcript, in a directory ultrafuzz itself configures via CLAUDE_CONFIG_DIR. Surfacing the assistant text on a failed attempt would turn an opaque failure into a self-explanatory one.
2. Immediate retries are the wrong response to this class of failure. There is already a precedent for distinguishing a waitable failure from a real one -- isTransientNpmRegistryFailure in npm-install-retry.ts, with a retry loop that waits. Nothing equivalent exists for agent invocation.

Possible directions, all of which look like design decisions rather than obvious fixes:

• stagger agent starts, or refresh the token once before fanning out
• give each concurrent agent its own provider home (on Linux the subscription credential is a file, so it is copyable -- though duplicating credentials has its own security posture question)
• classify this message as transient and retry with a wait, rather than immediately
• surface the agent transcript on failure so the operator is not left with Claude run failed

I have not opened a PR because I do not think it is my call which of those ultrafuzz should do, and a speculative fix here would be guessing at intent rather than fixing a defect.

Reproduction

1. Configure ClaudeAgent with auth = "subscription" and log in once so the credential lands in CLAUDE_CONFIG_DIR
2. Run a campaign whose first node fans out with max_parallel_agents: 4
3. Observe the node fail three times in under 30 seconds with Claude run failed
4. Read $CLAUDE_CONFIG_DIR/projects/<munged-worktree>/*.jsonl for the real message

Environment

Host

• macOS 26.6.2 (Darwin 25.6.0), arm64
• podman 6.0.2 client, applehv machine, 4 CPU / 8 GiB / 60 GB

Podman VM

• podman server 6.0.2, linux/arm64
• kernel 7.1.3-200.fc44.aarch64 (Fedora CoreOS 44), crun 1.28

Container

• base docker.io/library/node:24-bookworm (sha256:78b162211207872503ea9245188122b815150b9b4380e47a7c4a447332c01660)
• Debian GNU/Linux 12 (bookworm), aarch64, running as uid 1000
• --memory 7g --cpus 4
• CLAUDE_CONFIG_DIR=/claude on a podman named volume, mode 0700

Toolchain

• node v24.20.0, pnpm 11.1.1, bun 1.4.0, forge 1.8.1, git 2.39.5
• claude 2.1.258 (Claude Code), subscription auth (authMethod: claude.ai)

Ultrafuzz

• main (8b985b90) plus the fix from #1083, built in-image
• default audit profile, ClaudeAgent / claude-opus-5

Target

• morpho-org/metamorpho-v1.1 @ 89de526, submodules initialised

Start at the ClaudeAgent invocation boundary and compare it with the existing isTransientNpmRegistryFailure retry handling in npm-install-retry.ts. Reproduce the concurrent subscription-auth failure using max_parallel_agents: 4, then inspect the CLAUDE_CONFIG_DIR session transcripts for the discarded assistant message. Done requires an agreed approach for retry timing and failure reporting.