[Bug] formatRunError misclassifies any 'not found' error as an LLM 404 and fabricates provider/model details
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 119
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
Summary
formatRunError (packages/runtime-node/src/run.ts) classifies a run failure by
substring-matching the error text, and rewrites anything containing not found
or 404 into an LLM provider error — fabricating provider, model, and
baseURL details taken from resolved config rather than from whatever actually
failed:
if (/not found|404/i.test(error)) {
return [
'LLM 请求失败:404 Not Found。',
`请检查 provider/model/base-url:provider=${input.provider}, model=${input.model}, baseURL=${input.baseURL ?? '(default)'}`,
...
].join('\n');
}
Ordinary filesystem and tool errors contain "not found". They are then reported
to the user as an LLM misconfiguration that never occurred.
Reproduction
formatRunError('Cannot apply patch: file not found in context: src/hooks/useDebounce.ts',
{ provider: 'anthropic', model: 'claude-3-5-sonnet-20241022', debug: false });
returns:
LLM 请求失败:404 Not Found。
请检查 provider/model/base-url:provider=anthropic, model=claude-3-5-sonnet-20241022, baseURL=(default)
Anthropic provider 会请求 baseURL + /messages;请确认供应商支持 Anthropic Messages API。
No LLM call failed. The real error is the apply_patch skill's own message
(packages/core/src/skills/executor-skills.ts: Cannot apply patch: file not found in context: ${filePath}).
Also misfires on, e.g., Command failed: vitest run — 1 test not found.
It correctly passes through File src/a.ts does not exist... and
ENOENT: no such file or directory only because those happen not to contain the
literal token.
Impact
This actively misdirects debugging. Observed live: an ablation benchmark task
failed because a hook file was never created, and the run reported a
claude-3-5-sonnet-20241022 404 — a model the harness never configures. The
recorded telemetry for that same task was llmCalls: 10, llmFailures: 0, i.e.
every LLM call succeeded. I spent real time chasing a nonexistent provider
misconfiguration, and filed an incorrect root-cause issue on the strength of the
fabricated model name before catching it.
Anyone triaging from logs or a benchmark JSONL is exposed to the same trap, and
the fabricated provider/model/baseURL triple makes the wrong story look
well-evidenced.
Affected Area
runtime-node, CLI diagnostics
Environment
- OS: macOS (Darwin 25.4.0)
- Node: v24.18.0
- FrontAgent:
develop@ db42301 (v2.2.0)
Suggested direction
Classify by provenance, not by substring. Options, roughly in order of
preference:
- Have the LLM layer tag its own failures (an error subclass or a
kind: 'llm_request_failed'field) and letformatRunErrorswitch on that
tag; leave everything else untouched. - If substring matching must stay, require corroborating provider context
(e.g. an HTTP status attached by the provider client) before rewriting, and
never inventprovider/model/baseURLfor an error that carries none. - At minimum, preserve the original message alongside the hint instead of
replacing it — the current code discards the only accurate information.
Note formatRunError already returns the raw error unchanged when debug is
true, so the accurate text exists and is simply dropped in the default path.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with formatRunError in packages/runtime-node/src/run.ts and compare its substring matching with the error produced by packages/core/src/skills/executor-skills.ts. Trace how LLM failures are represented before choosing a provenance signal; done means ordinary tool and filesystem errors retain their original messages while genuine LLM failures still receive appropriate diagnostics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100