anthropics / anthropics/skills

mcp-builder: evaluation.py scores 0/N against any real MCP server (TextContent not JSON serializable, swallowed into fabricated tool errors)

Offen Anfängerfreundlich
#1,390 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
176k
Forks
20.9k
Ø Merge
7 Std. 21 Min.
Gemergte PRs (30 T.)
5

Beschreibung

## Summary

`skills/mcp-builder/scripts/evaluation.py` (the Phase-4 evaluation harness) silently fabricates a tool-execution error for **every tool call against every real MCP server**, so the driven agent sees nothing but errors and the evaluation scores 0/N. The failure is invisible unless you dig into what the model was actually shown.

## Root cause

`connections.py:70` returns `result.content` from `call_tool()` — which for a real MCP server is `list[mcp.types.TextContent]` (pydantic objects). `evaluation.py:117` then does:

```python
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```

Since the value is a `list`, it hits `json.dumps()`, which raises `TypeError: Object of type TextContent is not JSON serializable` on **every single call**. The surrounding `try/except` converts that into a fabricated `"Error executing tool ..."` message fed back to the model, which then (reasonably) answers NOT_FOUND / gives up on every task.

One-line repro against any live MCP server:

```python
import json
result_content = await session.call_tool(name, args) # list[TextContent]
json.dumps(result_content) # TypeError — this is what evaluation.py does on every call
```

## Fix

Serialize the content blocks' text instead of json.dumps-ing pydantic objects, e.g.:

```python
def _serialize_tool_result(tool_result):
if isinstance(tool_result, list):
return "\n".join(
block.text for block in tool_result if hasattr(block, "text")
)
return json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
```

We patched a local copy this way and the same 11-task suite went from 0/11 (all fabricated errors) to 11/11 with real tool traffic.

## Secondary issue (same script)

The hardcoded default model is retired: `evaluation.py:223` / `:324` default to `claude-3-7-sonnet-20250219`, which returns `404 not_found_error` — so a stock run dies before the serialization bug is even reachable. Suggest defaulting to a current alias or requiring `-m`.

Found while using the skill exactly as documented (Phase-4 evaluation of a streamable-HTTP MCP server, `-t http`).

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Read skills/mcp-builder/scripts/evaluation.py alongside connections.py, then run the documented Phase-4 evaluation against a streamable-HTTP MCP server and inspect the tool result handling. Done means real TextContent responses reach the model without fabricated errors, the 11-task suite no longer scores 0/N, and the stock model default no longer points to the retired model.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
testing-qa, tooling
Issue-Typ
Bug
Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
76/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.