anthropics / anthropics/claude-plugins-official

code-simplifier and agent-sdk-verifier-py assert verification results they never execute

Open
#4,785 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.3k
Forks
4.1k
Avg merge
2d 14h
Merged PRs (30d)
539

Description

## Summary

Two agents assert a verification result they never actually check. Both have the
tools to check it; the prompts simply never ask.

- `plugins/code-simplifier/agents/code-simplifier.md` (and its byte-identical twin
in `pr-review-toolkit`) edits code, then instructs the model to *"Ensure all
functionality remains unchanged"* — with no test, build, or typecheck anywhere
in the prompt.
- `plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md` emits a
`PASS | PASS WITH WARNINGS | FAIL` verdict on imports and syntax by reading the
source. Its TypeScript twin runs `npx tsc --noEmit` for the same claim.

---

## 1. `code-simplifier`: a refactoring agent with no verification step

`plugins/code-simplifier/agents/code-simplifier.md:43-50`

```md
Your refinement process:

1. Identify the recently modified code sections
2. Analyze for opportunities to improve elegance and consistency
3. Apply project-specific best practices and coding standards
4. Ensure all functionality remains unchanged <-- line 48
5. Verify the refined code is simpler and more maintainable <-- line 49
6. Document only significant changes that affect understanding
```

Step 4 is the agent's only safety property, and it is discharged by re-reading the
diff. Step 5 asks the agent to grade its own output on the single axis it was just
told to optimize.

The prompt never mentions running anything:

```console
$ grep -ciE '(test|build|lint|typecheck|tsc|npm|run )' \
plugins/code-simplifier/agents/code-simplifier.md \
plugins/pr-review-toolkit/agents/code-simplifier.md
plugins/code-simplifier/agents/code-simplifier.md:0
plugins/pr-review-toolkit/agents/code-simplifier.md:0
```

Neither file declares a `tools:` field, so both inherit the full tool set, `Bash`
included. The capability is there and unused.

The two files are the same agent — identical bodies, differing only in the
frontmatter `description`:

```console
$ diff <(sed -n '4,$p' plugins/code-simplifier/agents/code-simplifier.md) \
<(sed -n '40,$p' plugins/pr-review-toolkit/agents/code-simplifier.md)
$ echo $?
0
```

So the same lines appear at `code-simplifier/agents/code-simplifier.md:48-49` and
`pr-review-toolkit/agents/code-simplifier.md:84-85`.

### Why this is worth fixing beyond the wording

#208 (*"Code Simplifier Plugin alters strings"*) and #297 (*"code-simplifier
rewrites prompts and markdown files"*) both report the agent silently changing
behavior. Both proposed fixes add another prose prohibition to the prompt. But an
agent that had run the project's tests would have caught #208's case — rewritten
LLM prompt strings change output, and the failure is observable. The missing
execution gate is what lets "preserve functionality" violations reach the user in
the first place.

### Suggested fix

Replace steps 4 and 5 with a single step that produces evidence:

```md
4. Run the project's test suite and its typecheck/build command. Report the exact
commands and their outcomes. Report the refactor as safe only if the checks you
actually ran passed — never infer it. If the project has no such command, say so
explicitly and list the behaviors you could not verify.
5. Document only significant changes that affect understanding
```

The "never infer it" phrasing is not invented here — it is the pattern
`code-modernization/agents/uplift-migrator.md:42-45` already uses:

> Report the unit as built **only if the build you actually ran succeeded** — never
> infer or assume it. If you cannot run the build, say so and why; that is a valid
> result, "built" is not.

Applying it consistently to the agent that mutates code seems strictly better than
applying it only to the one that migrates it.

---

## 2. `agent-sdk-verifier-py`: verifies by reading, where the TS twin executes

The two SDK verifiers are built from the same template and issue the same verdict
format, but only one of them runs a check.

`agent-sdk-verifier-ts.md:101-104`:

```md
3. **Run Type Checking**:

- Execute `npx tsc --noEmit` to verify no type errors
- Report any compilation issues
```

`agent-sdk-verifier-py.md:95-99` — the corresponding step:

```md
3. **Validate Imports and Syntax**:

- Check that all imports are correct
- Look for obvious syntax errors
- Verify SDK is properly imported
```

And earlier, at `agent-sdk-verifier-py.md:37-41`:

```md
4. **Code Quality**:

- Check for basic syntax errors <-- line 39
- Verify imports are correct and available <-- line 40
```

`"available"` is the load-bearing word. Whether an import resolves is a property of
the installed environment, not of the source text — it is not observable by reading
the file. The agent then reports `PASS | PASS WITH WARNINGS | FAIL`
(`agent-sdk-verifier-py.md:106+`) on that basis.

Nothing in the file mandates executing anything. The one place a command appears,
it is offered as an alternative to reading two files:

```console
$ grep -nE '(python |pip |pytest|mypy|pyright|compileall)' \
plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md
15: - Verify `claude-agent-sdk` is installed (check requirements.txt, pyproject.toml, or pip list)
```

### Suggested fix

Mirror the TS version:

```md
3. **Validate Imports and Syntax**:

- Execute `python -m compileall -q .` to catch syntax errors
- Execute `python -c "import claude_agent_sdk"` inside the project's environment
to confirm the SDK actually resolves
- Report the exact commands and their outcomes; if the environment cannot be
activated, report that rather than inferring a result
```

Optionally add `mypy`/`pyright` if present in the project, to match the TS
verifier's type-checking coverage.

---

## Scope

I checked the other 31 agent prompts under `plugins/*/agents/` for the same
pattern — verification instructions with no execution behind them. These two are
the only cases. `uplift-migrator`, `test-engineer`, and `patch-generator` (which
requires a `git add -A` so a verifier judges a real staged diff) all get it right,
which is why the two above stand out rather than reading as a house style.

Happy to open a PR for either fix, though I understand external PRs are
auto-closed here, hence the issue.

## Environment

- Repo: `main` @ `d620fb7`
- Files read as-is from a fresh marketplace clone; line numbers are 1-indexed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read plugins/code-simplifier/agents/code-simplifier.md and its pr-review-toolkit twin, then compare agent-sdk-verifier-py.md with agent-sdk-verifier-ts.md. Check the proposed verification commands and the existing uplift-migrator wording first; done means the affected prompts require reporting commands actually run and their outcomes, including when verification is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.