Azure / Azure/azure-sdk-for-python
[azure-ai-projects] Trace-based evaluation cannot evaluate agents that use MCP tools — reconstructed messages have null tool_call fields
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
### Library / area
- `azure-ai-projects` (telemetry + Foundry trace-based evaluation)
- Affects: agents that use the **Foundry IQ / MCP tool** (`PromptAgentDefinition` + `MCPTool`)
### Problem
Trace-based evaluation (`data_source.type = "azure_ai_trace_data_source_preview"` with conversation-level evaluators using `{{item.messages}}`) **always errors out** for agents whose tool calls are emitted as `mcp_call` items. Every per-row result returns:
```
status: "error"
error.code: FAILED_EXECUTION
error.message: "'NoneType' object has no attribute 'items'"
```
(With the older query/response mapping the same data yields `(UserError) The 'name' field must be a string in tool_call content items.`)
### Reproduction
1. Create a `PromptAgentDefinition` whose `tools` include an `MCPTool` (e.g. a Foundry IQ knowledge base).
2. Enable tracing with `AIProjectInstrumentor().instrument(enable_content_recording=True)` and `configure_azure_monitor(...)`.
3. Drive a couple of `client.responses.create(..., extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}})` calls.
4. Wait for ingestion, collect `operation_Id`s from App Insights, then run the flow from `sample_multiturn_trace_evaluation_by_id.py`:
```python
data_source_config = {"type": "azure_ai_source", "scenario": "traces"}
testing_criteria = [
{"type": "azure_ai_evaluator", "name": "task_completion",
"evaluator_name": "builtin.task_completion",
"initialization_parameters": {"deployment_name": MODEL},
"data_mapping": {"messages": "{{item.messages}}"}},
{"type": "azure_ai_evaluator", "name": "coherence",
"evaluator_name": "builtin.coherence",
"initialization_parameters": {"deployment_name": MODEL},
"data_mapping": {"messages": "{{item.messages}}"}},
]
data_source = {
"type": "azure_ai_trace_data_source_preview",
"trace_source": {"type": "trace_id_source", "trace_ids": trace_ids},
}
client.evals.runs.create(eval_id=..., data_source=data_source,
extra_body={"evaluation_level": "conversation"})
```
### Actual output
`datasource_item.messages` contains assistant turns like:
```json
{"role": "assistant",
"content": [{"type": "tool_call",
"tool_call_id": null,
"name": null,
"arguments": null}]}
```
`name`, `tool_call_id`, and `arguments` are all `null`, so the evaluator crashes.
### Root cause (from the SDK source)
In [`sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/_responses_instrumentor.py`](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/_responses_instrumentor.py), `_emit_tool_call_event` has two formats:
```python
if _get_use_simple_tool_format() and tool_call.get("type") == "function_call":
# Flat, OTEL-compliant: {"type": "tool_call", "id": ..., "name": ..., "arguments": ...}
...
else:
# Nested: {"type": "tool_call", "content": {...}}
parts = [{"type": "tool_call", "content": tool_call}]
```
The simplified branch is gated on `type == "function_call"`. For `mcp_call` (and other `mcp_*` items handled later in `_add_tool_call_events`), the instrumentor always writes the nested form, where `name`/`arguments` live inside `content`.
The Foundry trace-evaluation service then flattens `tool_call` parts into the chat-completions shape it knows (`{"type": "tool_call", "tool_call_id", "name", "arguments"}`), but it doesn't appear to look inside `content` for `mcp_call` items, so it writes nulls — causing the evaluator failure.
### Expected
For agents with MCP tools, trace-based evaluation should reconstruct `messages` so that:
- `tool_call.name` matches what `_add_tool_call_events` captures for `mcp_call` (`output_item.name`, e.g. `knowledge_base_retrieve`)
- `tool_call.arguments` matches `output_item.arguments`
- `tool_call_id` matches `output_item.id`
…and the corresponding `tool` message reconstruction should pair correctly.
### Suggested fix (one of)
1. **Service-side** (trace-eval reconstruction): when flattening a `tool_call` part whose `content.type` starts with `mcp_`, lift `content.name`/`content.arguments`/`content.id` to the top-level OTEL fields, identical to the `function_call` path.
2. **SDK-side** (`_emit_tool_call_event`): extend the simple-format branch to also cover `mcp_call` so spans store `name`/`arguments` at the top level.
(1) is preferable because it also fixes traces already in App Insights.
### Environment
- `azure-ai-projects` 2.x (reproduced on current main as of 2026-06)
- Python 3.11, Linux (devcontainer)
- Repro stack: Foundry IQ knowledge base + `MCPTool` + `PromptAgentDefinition`, drives via `client.responses.create(...)` with `agent_reference`
- Tracing: `azure-monitor-opentelemetry` + `AIProjectInstrumentor`
- Eval samples cross-checked: `sample_multiturn_trace_evaluation_by_id.py`, `sample_multiturn_trace_evaluation_agent_filter.py`, `sample_multiturn_conversation_evaluation.py` (none cover MCP-tool agents)
### Workaround
None — all conversation-level trace evaluators fail with the same error as long as the trace contains any `mcp_call` assistant turn. Dataset-based and agent-target (`azure_ai_target_completions`) evaluations are unaffected.
Contributor guide
Assessment
This issue has not been assessed yet.