anthropics / anthropics/skills
mcp-builder: evaluation.py scores 0/N against any real MCP server (TextContent not JSON serializable, swallowed into fabricated tool errors)
- Ngôn ngữ chính
- Python
- Star
- 176k
- Fork
- 20.8k
- Merge trung bình
- 7 giờ 21 phút
- Pull request đã merge (30 ngày)
- 5
Mô tả
## Summary
`skills/mcp-builder/scripts/evaluation.py` (the Phase-4 evaluation harness) silently fabricates a tool-execution error for **every tool call against every real MCP server**, so the driven agent sees nothing but errors and the evaluation scores 0/N. The failure is invisible unless you dig into what the model was actually shown.
## Root cause
`connections.py:70` returns `result.content` from `call_tool()` — which for a real MCP server is `list[mcp.types.TextContent]` (pydantic objects). `evaluation.py:117` then does:
```python
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```
Since the value is a `list`, it hits `json.dumps()`, which raises `TypeError: Object of type TextContent is not JSON serializable` on **every single call**. The surrounding `try/except` converts that into a fabricated `"Error executing tool ..."` message fed back to the model, which then (reasonably) answers NOT_FOUND / gives up on every task.
One-line repro against any live MCP server:
```python
import json
result_content = await session.call_tool(name, args) # list[TextContent]
json.dumps(result_content) # TypeError — this is what evaluation.py does on every call
```
## Fix
Serialize the content blocks' text instead of json.dumps-ing pydantic objects, e.g.:
```python
def _serialize_tool_result(tool_result):
if isinstance(tool_result, list):
return "\n".join(
block.text for block in tool_result if hasattr(block, "text")
)
return json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```
We patched a local copy this way and the same 11-task suite went from 0/11 (all fabricated errors) to 11/11 with real tool traffic.
## Secondary issue (same script)
The hardcoded default model is retired: `evaluation.py:223` / `:324` default to `claude-3-7-sonnet-20250219`, which returns `404 not_found_error` — so a stock run dies before the serialization bug is even reachable. Suggest defaulting to a current alias or requiring `-m`.
Found while using the skill exactly as documented (Phase-4 evaluation of a streamable-HTTP MCP server, `-t http`).
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.