anthropics / anthropics/claude-plugins-official

code-simplifier and agent-sdk-verifier-py assert verification results they never execute

Aperta
#4,785 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
36.3k
Fork
4.1k
Merge medio
2g 14h
PR unite (30g)
539

Descrizione

## Summary

Two agents assert a verification result they never actually check. Both have the
tools to check it; the prompts simply never ask.

- `plugins/code-simplifier/agents/code-simplifier.md` (and its byte-identical twin
in `pr-review-toolkit`) edits code, then instructs the model to *"Ensure all
functionality remains unchanged"* — with no test, build, or typecheck anywhere
in the prompt.
- `plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md` emits a
`PASS | PASS WITH WARNINGS | FAIL` verdict on imports and syntax by reading the
source. Its TypeScript twin runs `npx tsc --noEmit` for the same claim.

---

## 1. `code-simplifier`: a refactoring agent with no verification step

`plugins/code-simplifier/agents/code-simplifier.md:43-50`

```md
Your refinement process:

1. Identify the recently modified code sections
2. Analyze for opportunities to improve elegance and consistency
3. Apply project-specific best practices and coding standards
4. Ensure all functionality remains unchanged <-- line 48
5. Verify the refined code is simpler and more maintainable <-- line 49
6. Document only significant changes that affect understanding
```

Step 4 is the agent's only safety property, and it is discharged by re-reading the
diff. Step 5 asks the agent to grade its own output on the single axis it was just
told to optimize.

The prompt never mentions running anything:

```console
$ grep -ciE '(test|build|lint|typecheck|tsc|npm|run )' \
plugins/code-simplifier/agents/code-simplifier.md \
plugins/pr-review-toolkit/agents/code-simplifier.md
plugins/code-simplifier/agents/code-simplifier.md:0
plugins/pr-review-toolkit/agents/code-simplifier.md:0
```

Neither file declares a `tools:` field, so both inherit the full tool set, `Bash`
included. The capability is there and unused.

The two files are the same agent — identical bodies, differing only in the
frontmatter `description`:

```console
$ diff <(sed -n '4,$p' plugins/code-simplifier/agents/code-simplifier.md) \
<(sed -n '40,$p' plugins/pr-review-toolkit/agents/code-simplifier.md)
$ echo $?
0
```

So the same lines appear at `code-simplifier/agents/code-simplifier.md:48-49` and
`pr-review-toolkit/agents/code-simplifier.md:84-85`.

### Why this is worth fixing beyond the wording

#208 (*"Code Simplifier Plugin alters strings"*) and #297 (*"code-simplifier
rewrites prompts and markdown files"*) both report the agent silently changing
behavior. Both proposed fixes add another prose prohibition to the prompt. But an
agent that had run the project's tests would have caught #208's case — rewritten
LLM prompt strings change output, and the failure is observable. The missing
execution gate is what lets "preserve functionality" violations reach the user in
the first place.

### Suggested fix

Replace steps 4 and 5 with a single step that produces evidence:

```md
4. Run the project's test suite and its typecheck/build command. Report the exact
commands and their outcomes. Report the refactor as safe only if the checks you
actually ran passed — never infer it. If the project has no such command, say so
explicitly and list the behaviors you could not verify.
5. Document only significant changes that affect understanding
```

The "never infer it" phrasing is not invented here — it is the pattern
`code-modernization/agents/uplift-migrator.md:42-45` already uses:

> Report the unit as built **only if the build you actually ran succeeded** — never
> infer or assume it. If you cannot run the build, say so and why; that is a valid
> result, "built" is not.

Applying it consistently to the agent that mutates code seems strictly better than
applying it only to the one that migrates it.

---

## 2. `agent-sdk-verifier-py`: verifies by reading, where the TS twin executes

The two SDK verifiers are built from the same template and issue the same verdict
format, but only one of them runs a check.

`agent-sdk-verifier-ts.md:101-104`:

```md
3. **Run Type Checking**:

- Execute `npx tsc --noEmit` to verify no type errors
- Report any compilation issues
```

`agent-sdk-verifier-py.md:95-99` — the corresponding step:

```md
3. **Validate Imports and Syntax**:

- Check that all imports are correct
- Look for obvious syntax errors
- Verify SDK is properly imported
```

And earlier, at `agent-sdk-verifier-py.md:37-41`:

```md
4. **Code Quality**:

- Check for basic syntax errors <-- line 39
- Verify imports are correct and available <-- line 40
```

`"available"` is the load-bearing word. Whether an import resolves is a property of
the installed environment, not of the source text — it is not observable by reading
the file. The agent then reports `PASS | PASS WITH WARNINGS | FAIL`
(`agent-sdk-verifier-py.md:106+`) on that basis.

Nothing in the file mandates executing anything. The one place a command appears,
it is offered as an alternative to reading two files:

```console
$ grep -nE '(python |pip |pytest|mypy|pyright|compileall)' \
plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md
15: - Verify `claude-agent-sdk` is installed (check requirements.txt, pyproject.toml, or pip list)
```

### Suggested fix

Mirror the TS version:

```md
3. **Validate Imports and Syntax**:

- Execute `python -m compileall -q .` to catch syntax errors
- Execute `python -c "import claude_agent_sdk"` inside the project's environment
to confirm the SDK actually resolves
- Report the exact commands and their outcomes; if the environment cannot be
activated, report that rather than inferring a result
```

Optionally add `mypy`/`pyright` if present in the project, to match the TS
verifier's type-checking coverage.

---

## Scope

I checked the other 31 agent prompts under `plugins/*/agents/` for the same
pattern — verification instructions with no execution behind them. These two are
the only cases. `uplift-migrator`, `test-engineer`, and `patch-generator` (which
requires a `git add -A` so a verifier judges a real staged diff) all get it right,
which is why the two above stand out rather than reading as a house style.

Happy to open a PR for either fix, though I understand external PRs are
auto-closed here, hence the issue.

## Environment

- Repo: `main` @ `d620fb7`
- Files read as-is from a fresh marketplace clone; line numbers are 1-indexed.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Read plugins/code-simplifier/agents/code-simplifier.md and its pr-review-toolkit twin, then compare agent-sdk-verifier-py.md with agent-sdk-verifier-ts.md. Check the proposed verification commands and the existing uplift-migrator wording first; done means the affected prompts require reporting commands actually run and their outcomes, including when verification is unavailable.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, typescript
Ambito
testing-qa, tooling
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
72/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.