anthropics / anthropics/skills

mcp-builder: evaluation.py scores 0/N against any real MCP server (TextContent not JSON serializable, swallowed into fabricated tool errors)

Open Beginner friendly
#1,390 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
176k
Forks
20.9k
Avg merge
7h 21m
Merged PRs (30d)
5

Description

## Summary

`skills/mcp-builder/scripts/evaluation.py` (the Phase-4 evaluation harness) silently fabricates a tool-execution error for **every tool call against every real MCP server**, so the driven agent sees nothing but errors and the evaluation scores 0/N. The failure is invisible unless you dig into what the model was actually shown.

## Root cause

`connections.py:70` returns `result.content` from `call_tool()` — which for a real MCP server is `list[mcp.types.TextContent]` (pydantic objects). `evaluation.py:117` then does:

```python
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```

Since the value is a `list`, it hits `json.dumps()`, which raises `TypeError: Object of type TextContent is not JSON serializable` on **every single call**. The surrounding `try/except` converts that into a fabricated `"Error executing tool ..."` message fed back to the model, which then (reasonably) answers NOT_FOUND / gives up on every task.

One-line repro against any live MCP server:

```python
import json
result_content = await session.call_tool(name, args) # list[TextContent]
json.dumps(result_content) # TypeError — this is what evaluation.py does on every call
```

## Fix

Serialize the content blocks' text instead of json.dumps-ing pydantic objects, e.g.:

```python
def _serialize_tool_result(tool_result):
if isinstance(tool_result, list):
return "\n".join(
block.text for block in tool_result if hasattr(block, "text")
)
return json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```

We patched a local copy this way and the same 11-task suite went from 0/11 (all fabricated errors) to 11/11 with real tool traffic.

## Secondary issue (same script)

The hardcoded default model is retired: `evaluation.py:223` / `:324` default to `claude-3-7-sonnet-20250219`, which returns `404 not_found_error` — so a stock run dies before the serialization bug is even reachable. Suggest defaulting to a current alias or requiring `-m`.

Found while using the skill exactly as documented (Phase-4 evaluation of a streamable-HTTP MCP server, `-t http`).

Contributor guide

No contributing guide indexed for this repository

Research direction

Read skills/mcp-builder/scripts/evaluation.py alongside connections.py, then run the documented Phase-4 evaluation against a streamable-HTTP MCP server and inspect the tool result handling. Done means real TextContent responses reach the model without fabricated errors, the 11-task suite no longer scores 0/N, and the stock model default no longer points to the retired model.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.