MCPJam / MCPJam/inspector

Swarms: a run that never executed still renders fabricated findings

Open
#5,188 0 comments 0 reactions 0 assignees View on GitHub
bug Highest Priority
Dominant language
TypeScript
Stars
2.2k
Forks
283
Avg merge
11h 47m
Merged PRs (30d)
737

Description

Split out of #5184. That PR fixed the outage; this is the reason the outage was misread as a product failure for three days.

## What happens

When a swarm run is refused before any session executes — the identity `400` from #5184, a provider rate limit, a connection failure — Findings, Insights and Sessions do not say so. They render findings derived from sessions that never produced a transcript:

- the rubric `Final message non-empty` fails, graded against a transcript that was never written
- Insights shows `Not analyzed — N · 100%`
- the journey diagnostic reports friction at **connection**, and personas are marked `UNEASY` / `STALLED`
- Sessions shows "No messages in this session" and `Session 1 (no preview)`

None of that came from the server under test. A user reads it as "the MCP server under test has UX problems at connection", which is a claim the product has no evidence for.

## Why it matters

This is what made the #5184 outage read as "Swarms is broken" rather than "one endpoint returned 400 on every turn". **While that outage was live, no surface stated the real error at all** — it took three days to place because every tab that spoke described the server under test instead.

(An earlier revision of this issue credited the Run tab with telling the truth throughout. That was wrong: the Run tab landed in #5151, merged 2026-09-16T05:35Z, about three hours **after** the outage was reported at 02:41Z. It has since been asked for removal as out-of-scope UI, which makes this issue more urgent rather than less — without it a refused run again has nowhere to say why it did not execute.)

The failure mode outlives that specific bug: any future refusal produces the same fabricated output.

## Where

Findings derivation is entirely client-side with no queries — `client/src/components/swarms/findings/findings-derivation.ts` builds stages from `wave.runs` + `waveSignals`, and `getSwarmOverview` reads run rows only.

Clustering drops sessions with an unreadable or empty transcript from `prepared` (`mcpjam-backend convex/scenarioClusters.ts`, `prepareTranscripts`), then `runSignalDimensionPasses` clears the downstream cluster ids — which is what produces `Not analyzed — 100%` rather than a "not run" state.

## Suggested shape

A run whose attempts are all terminal-without-execution needs to propagate that as a first-class state the three tabs render, instead of each one independently inferring a product finding from absent data. Findings should say the run did not execute and why; Insights and Sessions should defer to that rather than describing an empty cohort.

One nuance worth preserving: goals that did not run currently surface as `UNSCORED` under a rate limit, which is already the right shape. The identity-failure path did not get that treatment. Whatever lands should make the honest state the default rather than something each path opts into.

## Verified on

staging, run `c35bfd32-48bf-4691-9226-726ce3c02636` (post-fix, 7 of 15 with 8 rate-limited) and the original prod report on `f2d291df-067a-4d00-ae0c-4175bb2e1449`.

Contributor guide

Open the contributing guide

Research direction

Start with client/src/components/swarms/findings/findings-derivation.ts and trace how wave.runs and waveSignals reach Findings, Insights, and Sessions. Read mcpjam-backend convex/scenarioClusters.ts, especially prepareTranscripts and runSignalDimensionPasses, then compare the verified staging run with the existing UNSCORED rate-limit behavior. Done means terminal-without-execution runs show their error honestly across all three tabs without fabricated findings or diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.