anthropics / anthropics/skills
mcp-builder: evaluation.py scores 0/N against any real MCP server (TextContent not JSON serializable, swallowed into fabricated tool errors)
- Vorherrschende Sprache
- Python
- Sterne
- 176k
- Forks
- 20.9k
- Ø Merge
- 7 Std. 21 Min.
- Gemergte PRs (30 T.)
- 5
Beschreibung
## Summary
`skills/mcp-builder/scripts/evaluation.py` (the Phase-4 evaluation harness) silently fabricates a tool-execution error for **every tool call against every real MCP server**, so the driven agent sees nothing but errors and the evaluation scores 0/N. The failure is invisible unless you dig into what the model was actually shown.
## Root cause
`connections.py:70` returns `result.content` from `call_tool()` — which for a real MCP server is `list[mcp.types.TextContent]` (pydantic objects). `evaluation.py:117` then does:
```python
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```
Since the value is a `list`, it hits `json.dumps()`, which raises `TypeError: Object of type TextContent is not JSON serializable` on **every single call**. The surrounding `try/except` converts that into a fabricated `"Error executing tool ..."` message fed back to the model, which then (reasonably) answers NOT_FOUND / gives up on every task.
One-line repro against any live MCP server:
```python
import json
result_content = await session.call_tool(name, args) # list[TextContent]
json.dumps(result_content) # TypeError — this is what evaluation.py does on every call
```
## Fix
Serialize the content blocks' text instead of json.dumps-ing pydantic objects, e.g.:
```python
def _serialize_tool_result(tool_result):
if isinstance(tool_result, list):
return "\n".join(
block.text for block in tool_result if hasattr(block, "text")
)
return json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```
We patched a local copy this way and the same 11-task suite went from 0/11 (all fabricated errors) to 11/11 with real tool traffic.
## Secondary issue (same script)
The hardcoded default model is retired: `evaluation.py:223` / `:324` default to `claude-3-7-sonnet-20250219`, which returns `404 not_found_error` — so a stock run dies before the serialization bug is even reachable. Suggest defaulting to a current alias or requiring `-m`.
Found while using the skill exactly as documented (Phase-4 evaluation of a streamable-HTTP MCP server, `-t http`).
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Read skills/mcp-builder/scripts/evaluation.py alongside connections.py, then run the documented Phase-4 evaluation against a streamable-HTTP MCP server and inspect the tool result handling. Done means real TextContent responses reach the model without fabricated errors, the 11-task suite no longer scores 0/N, and the stock model default no longer points to the retired model.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- testing-qa, tooling
- Issue-Typ
- Bug
- Schwierigkeit
- 2/5
- Geschätzter Aufwand
- 1-3 Stunden
- Aktivitätsstatus
- Aktiv
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 76/100