[Bug] Guard syntax check passes prose-as-code — TS1127 corruption reaches disk unflagged
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 119
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
Summary
checkSyntaxValidity for TypeScript/JavaScript is not a parser — it is bracket
matching plus a small set of regex "common error patterns"
(packages/hallucination-guard/src/checks/syntax-validity.ts,
checkJavaScriptSyntax → checkBrackets + checkCommonPatterns).
Consequence: natural-language prose passes the syntax check, because prose
has balanced brackets and matches none of the error patterns. This is not
hypothetical — it is the dominant real failure mode observed in a benchmark run
today.
When the model answers with a clarifying question instead of code, the agent
writes that answer into the target source file. tsc then reports
TS1127: Invalid character, but the guard has already passed the content, so
nothing is blocked and nothing is rolled back.
Reproduction
Measured directly against the guard, using the two files an agent run actually
produced:
formatDate.ts guard.validateCode pass = true | blockedBy = null
head = "我需要先澄清一下:您提供的代码中没有具体说明要做什么修改。..."
TodoList.tsx guard.validateCode pass = true | blockedBy = null
head = "无法完成。您提供了原始代码和一般性的修改要求..."
Both files are pure Chinese prose. Both were written to disk. tsc --noEmit
on them:
src/utils/formatDate.ts(1,9): error TS1127: Invalid character.
src/components/TodoList.tsx(1,5): error TS1127: Invalid character.
Minimal repro:
const guard = new HallucinationGuard({ projectRoot: '/tmp' });
await guard.validateCode('这是一段说明文字,不是代码。', 'typescript', 'a.ts');
// => { pass: true }
Impact
In a 5-task smoke run of the ablation benchmark (post-#402), 2 of 5 tasks failed
with exactly this signature, and validation_failed was 0 for both — the guard
never objected. PR #402 makes the executor block writes that the guard flags
and emit validation_failed; it cannot help when the checker itself returns
pass: true. So this is now the binding constraint on interception rate, not
the executor plumbing.
Note the related weakness in cleanGeneratedCode
(packages/core/src/llm/code-generation.ts): its fence stripper is
/^```[\w]*\n/m with no g flag, so it removes only the first fence. Multi-block
model output still leaves inner fences in the file — another TS1127 source that
bracket matching will not catch.
Affected Area
hallucination-guard, executor, benchmarks
Environment
- OS: macOS (Darwin 25.4.0)
- Node: v24.18.0
- pnpm: 9.0.0
- FrontAgent:
develop@ db42301 (v2.2.0) + PR #402 branch - Model under test:
claude-haiku-4-5via Claude Code CLI text backend
Suggested direction
A real parse is the only reliable check here. The constraint is that
packages/hallucination-guard intentionally ships with zero runtime
dependencies for CLI distribution, so adding @babel/parser or typescript as
a dependency is a maintainer-level call, not something to slip in.
Cheaper options that stay dependency-free and would have caught both cases:
- Reject content whose first non-blank, non-comment line does not look like
code (noimport/export/const/function/class/type/interface/{/<).
This is the same heuristiccleanGeneratedCodealready uses to find
codeStartIndex— it just is not used as a rejection signal. - Reject a
.ts/.tsxfile whose ratio of CJK (or generally non-ASCII
non-comment) characters outside string literals exceeds a small threshold. - Fix the fence stripper to be global, and reject any residual ``` in output.
Happy to implement whichever direction maintainers prefer.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/hallucination-guard/src/checks/syntax-validity.ts, especially checkJavaScriptSyntax, checkBrackets, and checkCommonPatterns, then inspect cleanGeneratedCode in packages/core/src/llm/code-generation.ts. Run the minimal validateCode reproduction and compare it with tsc --noEmit on the affected files. Done means prose and residual multi-block fences are intercepted rather than written as source, with the chosen dependency-free or parser-based direction agreed by maintainers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- devtools, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100