anthropics / anthropics/skills
mcp-builder: evaluation.py scores 0/N against any real MCP server (TextContent not JSON serializable, swallowed into fabricated tool errors)
- Dominant language
- Python
- Stars
- 176k
- Forks
- 20.9k
- Avg merge
- 7h 21m
- Merged PRs (30d)
- 5
Description
## Summary
`skills/mcp-builder/scripts/evaluation.py` (the Phase-4 evaluation harness) silently fabricates a tool-execution error for **every tool call against every real MCP server**, so the driven agent sees nothing but errors and the evaluation scores 0/N. The failure is invisible unless you dig into what the model was actually shown.
## Root cause
`connections.py:70` returns `result.content` from `call_tool()` — which for a real MCP server is `list[mcp.types.TextContent]` (pydantic objects). `evaluation.py:117` then does:
```python
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```
Since the value is a `list`, it hits `json.dumps()`, which raises `TypeError: Object of type TextContent is not JSON serializable` on **every single call**. The surrounding `try/except` converts that into a fabricated `"Error executing tool ..."` message fed back to the model, which then (reasonably) answers NOT_FOUND / gives up on every task.
One-line repro against any live MCP server:
```python
import json
result_content = await session.call_tool(name, args) # list[TextContent]
json.dumps(result_content) # TypeError — this is what evaluation.py does on every call
```
## Fix
Serialize the content blocks' text instead of json.dumps-ing pydantic objects, e.g.:
```python
def _serialize_tool_result(tool_result):
if isinstance(tool_result, list):
return "\n".join(
block.text for block in tool_result if hasattr(block, "text")
)
return json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```
We patched a local copy this way and the same 11-task suite went from 0/11 (all fabricated errors) to 11/11 with real tool traffic.
## Secondary issue (same script)
The hardcoded default model is retired: `evaluation.py:223` / `:324` default to `claude-3-7-sonnet-20250219`, which returns `404 not_found_error` — so a stock run dies before the serialization bug is even reachable. Suggest defaulting to a current alias or requiring `-m`.
Found while using the skill exactly as documented (Phase-4 evaluation of a streamable-HTTP MCP server, `-t http`).
Contributor guide
No contributing guide indexed for this repository
Research direction
Read skills/mcp-builder/scripts/evaluation.py alongside connections.py, then run the documented Phase-4 evaluation against a streamable-HTTP MCP server and inspect the tool result handling. Done means real TextContent responses reach the model without fabricated errors, the 11-task suite no longer scores 0/N, and the stock model default no longer points to the retired model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100