ceilf6 / ceilf6/FrontAgent

[Bug] formatRunError misclassifies any 'not found' error as an LLM 404 and fabricates provider/model details

Open
#408 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
119
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Summary

formatRunError (packages/runtime-node/src/run.ts) classifies a run failure by
substring-matching the error text, and rewrites anything containing not found
or 404 into an LLM provider error — fabricating provider, model, and
baseURL details taken from resolved config rather than from whatever actually
failed:

if (/not found|404/i.test(error)) {
  return [
    'LLM 请求失败:404 Not Found。',
    `请检查 provider/model/base-url:provider=${input.provider}, model=${input.model}, baseURL=${input.baseURL ?? '(default)'}`,
    ...
  ].join('\n');
}

Ordinary filesystem and tool errors contain "not found". They are then reported
to the user as an LLM misconfiguration that never occurred.

Reproduction

formatRunError('Cannot apply patch: file not found in context: src/hooks/useDebounce.ts',
               { provider: 'anthropic', model: 'claude-3-5-sonnet-20241022', debug: false });

returns:

LLM 请求失败:404 Not Found。
请检查 provider/model/base-url:provider=anthropic, model=claude-3-5-sonnet-20241022, baseURL=(default)
Anthropic provider 会请求 baseURL + /messages;请确认供应商支持 Anthropic Messages API。

No LLM call failed. The real error is the apply_patch skill's own message
(packages/core/src/skills/executor-skills.ts: Cannot apply patch: file not found in context: ${filePath}).

Also misfires on, e.g., Command failed: vitest run — 1 test not found.
It correctly passes through File src/a.ts does not exist... and
ENOENT: no such file or directory only because those happen not to contain the
literal token.

Impact

This actively misdirects debugging. Observed live: an ablation benchmark task
failed because a hook file was never created, and the run reported a
claude-3-5-sonnet-20241022 404 — a model the harness never configures. The
recorded telemetry for that same task was llmCalls: 10, llmFailures: 0, i.e.
every LLM call succeeded. I spent real time chasing a nonexistent provider
misconfiguration, and filed an incorrect root-cause issue on the strength of the
fabricated model name before catching it.

Anyone triaging from logs or a benchmark JSONL is exposed to the same trap, and
the fabricated provider/model/baseURL triple makes the wrong story look
well-evidenced.

Affected Area

runtime-node, CLI diagnostics

Environment

  • OS: macOS (Darwin 25.4.0)
  • Node: v24.18.0
  • FrontAgent: develop @ db42301 (v2.2.0)

Suggested direction

Classify by provenance, not by substring. Options, roughly in order of
preference:

  1. Have the LLM layer tag its own failures (an error subclass or a
    kind: 'llm_request_failed' field) and let formatRunError switch on that
    tag; leave everything else untouched.
  2. If substring matching must stay, require corroborating provider context
    (e.g. an HTTP status attached by the provider client) before rewriting, and
    never invent provider/model/baseURL for an error that carries none.
  3. At minimum, preserve the original message alongside the hint instead of
    replacing it — the current code discards the only accurate information.

Note formatRunError already returns the raw error unchanged when debug is
true, so the accurate text exists and is simply dropped in the default path.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with formatRunError in packages/runtime-node/src/run.ts and compare its substring matching with the error produced by packages/core/src/skills/executor-skills.ts. Trace how LLM failures are represented before choosing a provenance signal; done means ordinary tool and filesystem errors retain their original messages while genuine LLM failures still receive appropriate diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.