ceilf6 / ceilf6/FrontAgent

[Bug] Guard syntax check passes prose-as-code — TS1127 corruption reaches disk unflagged

Open
#406 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
119
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Summary

checkSyntaxValidity for TypeScript/JavaScript is not a parser — it is bracket
matching plus a small set of regex "common error patterns"
(packages/hallucination-guard/src/checks/syntax-validity.ts,
checkJavaScriptSyntaxcheckBrackets + checkCommonPatterns).

Consequence: natural-language prose passes the syntax check, because prose
has balanced brackets and matches none of the error patterns. This is not
hypothetical — it is the dominant real failure mode observed in a benchmark run
today.

When the model answers with a clarifying question instead of code, the agent
writes that answer into the target source file. tsc then reports
TS1127: Invalid character, but the guard has already passed the content, so
nothing is blocked and nothing is rolled back.

Reproduction

Measured directly against the guard, using the two files an agent run actually
produced:

formatDate.ts    guard.validateCode pass = true | blockedBy = null
                 head = "我需要先澄清一下:您提供的代码中没有具体说明要做什么修改。..."
TodoList.tsx     guard.validateCode pass = true | blockedBy = null
                 head = "无法完成。您提供了原始代码和一般性的修改要求..."

Both files are pure Chinese prose. Both were written to disk. tsc --noEmit
on them:

src/utils/formatDate.ts(1,9): error TS1127: Invalid character.
src/components/TodoList.tsx(1,5): error TS1127: Invalid character.

Minimal repro:

const guard = new HallucinationGuard({ projectRoot: '/tmp' });
await guard.validateCode('这是一段说明文字,不是代码。', 'typescript', 'a.ts');
// => { pass: true }

Impact

In a 5-task smoke run of the ablation benchmark (post-#402), 2 of 5 tasks failed
with exactly this signature, and validation_failed was 0 for both — the guard
never objected. PR #402 makes the executor block writes that the guard flags
and emit validation_failed; it cannot help when the checker itself returns
pass: true. So this is now the binding constraint on interception rate, not
the executor plumbing.

Note the related weakness in cleanGeneratedCode
(packages/core/src/llm/code-generation.ts): its fence stripper is
/^```[\w]*\n/m with no g flag, so it removes only the first fence. Multi-block
model output still leaves inner fences in the file — another TS1127 source that
bracket matching will not catch.

Affected Area

hallucination-guard, executor, benchmarks

Environment

  • OS: macOS (Darwin 25.4.0)
  • Node: v24.18.0
  • pnpm: 9.0.0
  • FrontAgent: develop @ db42301 (v2.2.0) + PR #402 branch
  • Model under test: claude-haiku-4-5 via Claude Code CLI text backend

Suggested direction

A real parse is the only reliable check here. The constraint is that
packages/hallucination-guard intentionally ships with zero runtime
dependencies
for CLI distribution, so adding @babel/parser or typescript as
a dependency is a maintainer-level call, not something to slip in.

Cheaper options that stay dependency-free and would have caught both cases:

  1. Reject content whose first non-blank, non-comment line does not look like
    code (no import/export/const/function/class/type/interface/{/<).
    This is the same heuristic cleanGeneratedCode already uses to find
    codeStartIndex — it just is not used as a rejection signal.
  2. Reject a .ts/.tsx file whose ratio of CJK (or generally non-ASCII
    non-comment) characters outside string literals exceeds a small threshold.
  3. Fix the fence stripper to be global, and reject any residual ``` in output.

Happy to implement whichever direction maintainers prefer.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/hallucination-guard/src/checks/syntax-validity.ts, especially checkJavaScriptSyntax, checkBrackets, and checkCommonPatterns, then inspect cleanGeneratedCode in packages/core/src/llm/code-generation.ts. Run the minimal validateCode reproduction and compare it with tsc --noEmit on the affected files. Done means prose and residual multi-block fences are intercepted rather than written as source, with the chosen dependency-free or parser-based direction agreed by maintainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
devtools, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.