anthropics / anthropics/claude-plugins-official
code-simplifier and agent-sdk-verifier-py assert verification results they never execute
- Linguagem predominante
- Python
- Estrelas
- 36.3k
- Forks
- 4.1k
- Merge médio
- 2d 14h
- PRs com merge (30d)
- 539
Descrição
## Summary
Two agents assert a verification result they never actually check. Both have the
tools to check it; the prompts simply never ask.
- `plugins/code-simplifier/agents/code-simplifier.md` (and its byte-identical twin
in `pr-review-toolkit`) edits code, then instructs the model to *"Ensure all
functionality remains unchanged"* — with no test, build, or typecheck anywhere
in the prompt.
- `plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md` emits a
`PASS | PASS WITH WARNINGS | FAIL` verdict on imports and syntax by reading the
source. Its TypeScript twin runs `npx tsc --noEmit` for the same claim.
---
## 1. `code-simplifier`: a refactoring agent with no verification step
`plugins/code-simplifier/agents/code-simplifier.md:43-50`
```md
Your refinement process:
1. Identify the recently modified code sections
2. Analyze for opportunities to improve elegance and consistency
3. Apply project-specific best practices and coding standards
4. Ensure all functionality remains unchanged <-- line 48
5. Verify the refined code is simpler and more maintainable <-- line 49
6. Document only significant changes that affect understanding
```
Step 4 is the agent's only safety property, and it is discharged by re-reading the
diff. Step 5 asks the agent to grade its own output on the single axis it was just
told to optimize.
The prompt never mentions running anything:
```console
$ grep -ciE '(test|build|lint|typecheck|tsc|npm|run )' \
plugins/code-simplifier/agents/code-simplifier.md \
plugins/pr-review-toolkit/agents/code-simplifier.md
plugins/code-simplifier/agents/code-simplifier.md:0
plugins/pr-review-toolkit/agents/code-simplifier.md:0
```
Neither file declares a `tools:` field, so both inherit the full tool set, `Bash`
included. The capability is there and unused.
The two files are the same agent — identical bodies, differing only in the
frontmatter `description`:
```console
$ diff <(sed -n '4,$p' plugins/code-simplifier/agents/code-simplifier.md) \
<(sed -n '40,$p' plugins/pr-review-toolkit/agents/code-simplifier.md)
$ echo $?
0
```
So the same lines appear at `code-simplifier/agents/code-simplifier.md:48-49` and
`pr-review-toolkit/agents/code-simplifier.md:84-85`.
### Why this is worth fixing beyond the wording
#208 (*"Code Simplifier Plugin alters strings"*) and #297 (*"code-simplifier
rewrites prompts and markdown files"*) both report the agent silently changing
behavior. Both proposed fixes add another prose prohibition to the prompt. But an
agent that had run the project's tests would have caught #208's case — rewritten
LLM prompt strings change output, and the failure is observable. The missing
execution gate is what lets "preserve functionality" violations reach the user in
the first place.
### Suggested fix
Replace steps 4 and 5 with a single step that produces evidence:
```md
4. Run the project's test suite and its typecheck/build command. Report the exact
commands and their outcomes. Report the refactor as safe only if the checks you
actually ran passed — never infer it. If the project has no such command, say so
explicitly and list the behaviors you could not verify.
5. Document only significant changes that affect understanding
```
The "never infer it" phrasing is not invented here — it is the pattern
`code-modernization/agents/uplift-migrator.md:42-45` already uses:
> Report the unit as built **only if the build you actually ran succeeded** — never
> infer or assume it. If you cannot run the build, say so and why; that is a valid
> result, "built" is not.
Applying it consistently to the agent that mutates code seems strictly better than
applying it only to the one that migrates it.
---
## 2. `agent-sdk-verifier-py`: verifies by reading, where the TS twin executes
The two SDK verifiers are built from the same template and issue the same verdict
format, but only one of them runs a check.
`agent-sdk-verifier-ts.md:101-104`:
```md
3. **Run Type Checking**:
- Execute `npx tsc --noEmit` to verify no type errors
- Report any compilation issues
```
`agent-sdk-verifier-py.md:95-99` — the corresponding step:
```md
3. **Validate Imports and Syntax**:
- Check that all imports are correct
- Look for obvious syntax errors
- Verify SDK is properly imported
```
And earlier, at `agent-sdk-verifier-py.md:37-41`:
```md
4. **Code Quality**:
- Check for basic syntax errors <-- line 39
- Verify imports are correct and available <-- line 40
```
`"available"` is the load-bearing word. Whether an import resolves is a property of
the installed environment, not of the source text — it is not observable by reading
the file. The agent then reports `PASS | PASS WITH WARNINGS | FAIL`
(`agent-sdk-verifier-py.md:106+`) on that basis.
Nothing in the file mandates executing anything. The one place a command appears,
it is offered as an alternative to reading two files:
```console
$ grep -nE '(python |pip |pytest|mypy|pyright|compileall)' \
plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md
15: - Verify `claude-agent-sdk` is installed (check requirements.txt, pyproject.toml, or pip list)
```
### Suggested fix
Mirror the TS version:
```md
3. **Validate Imports and Syntax**:
- Execute `python -m compileall -q .` to catch syntax errors
- Execute `python -c "import claude_agent_sdk"` inside the project's environment
to confirm the SDK actually resolves
- Report the exact commands and their outcomes; if the environment cannot be
activated, report that rather than inferring a result
```
Optionally add `mypy`/`pyright` if present in the project, to match the TS
verifier's type-checking coverage.
---
## Scope
I checked the other 31 agent prompts under `plugins/*/agents/` for the same
pattern — verification instructions with no execution behind them. These two are
the only cases. `uplift-migrator`, `test-engineer`, and `patch-generator` (which
requires a `git add -A` so a verifier judges a real staged diff) all get it right,
which is why the two above stand out rather than reading as a house style.
Happy to open a PR for either fix, though I understand external PRs are
auto-closed here, hence the issue.
## Environment
- Repo: `main` @ `d620fb7`
- Files read as-is from a fresh marketplace clone; line numbers are 1-indexed.
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Read plugins/code-simplifier/agents/code-simplifier.md and its pr-review-toolkit twin, then compare agent-sdk-verifier-py.md with agent-sdk-verifier-ts.md. Check the proposed verification commands and the existing uplift-migrator wording first; done means the affected prompts require reporting commands actually run and their outcomes, including when verification is unavailable.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python, typescript
- Domínio
- testing-qa, tooling
- Tipo de issue
- Bug
- Dificuldade
- 3/5
- Tempo estimado
- 1-2 dias
- Status de atividade
- Pouca atividade
- Clareza
- Claramente especificada
- Facilidade para iniciantes
- 72/100