Azure / Azure/azure-sdk-for-python
Telemetry Gap - Foundry Agent with Agent Knowledge Lacks Observability Details
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 202
Description
When using an Azure AI Foundry agent with an MCP Knowledge Base tool (`mcp_kb_knowledgebase842_8a5nl.knowledge_base_retrieve`), the telemetry/tracing data captured in Application Insights has two critical gaps:
1. **Missing Tool Call Response**: The tool's input parameters and retrieved content are not visible in telemetry
2. **Missing MCP Tool Approval Request**: The `mcp_approval_request` event that should be captured when the agent requires tool consent is not visible in the traces
This makes it impossible to debug, audit, and understand the agent's knowledge retrieval behavior and approval workflow.
## Current Behavior
In the Application Insights End-to-End Transaction view, the `mcp_kb_knowledgebase842_8a5nl.knowledge_base_retrieve` → `execute_tool` span shows:
| Property | Value |
|----------|-------|
| EVENT | mcp_kb_knowledgebase842_8a5nl.knowledge_base_retrieve execute_tool |
| RES. | 0 |
| DURATION | 98.4 ms |
| Messages (Tool) | (empty or minimal content) |
### What's Missing
1. **Tool Input Parameters**: The query, filters, or any parameters passed to the Knowledge Base tool are not visible
2. **Retrieved Content**: The documents/results returned from the knowledge base are not captured
3. **MCP Approval Request**: No trace of the `mcp_approval_request` event that should be generated when the tool requires user consent
4. **Approval Response**: No visibility into whether approval was granted or denied
5. **Knowledge Base Metadata**: No visibility into which knowledge base was queried, number of results, etc.
## Expected Behavior
The telemetry for MCP Knowledge Base tools should include:
1. **Tool Input**:
- Query text sent to the knowledge base
2. **Tool Output**:
- Retrieved document snippets/content
- Number of documents returned
- Relevance scores (if available)
3. **MCP Approval Flow**:
- `mcp_approval_request` event with tool name and request ID
- Approval status (approved/denied)
- Timestamp of approval response
4. **Metadata**:
- Knowledge base endpoint/connection
- Request latency breakdown
## Reproduction Steps
1. Create an Azure AI Foundry agent with an MCP Knowledge Base tool configured
2. Run the agent with a query that triggers the knowledge base tool:
```python
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.monitor.opentelemetry import configure_azure_monitor
# Configure Azure Monitor for observability
configure_azure_monitor(connection_string=os.environ.get("APPLICATION_INSIGHTS_CNX_STR"))
with DefaultAzureCredential() as credential:
with AIProjectClient(endpoint=endpoint, credential=credential) as project_client:
openai_client = project_client.get_openai_client()
stream_response = openai_client.responses.create(
stream=True,
input="Hello! Get all tests from catalog.",
extra_body={"agent": {"name": "test-kb-tool", "type": "agent_reference"}},
)
for event in stream_response:
# Handle mcp_approval_request events
if event.type == "response.output_item.done":
if event.item.type == "mcp_approval_request":
# This event IS received in the SDK but NOT captured in telemetry
pass
```
3. Check Application Insights → End-to-End Transaction
4. Observe that:
- The `mcp_kb_*.knowledge_base_retrieve` → `execute_tool` span has empty/minimal content
- No `mcp_approval_request` event is visible in the trace
## Evidence
**Operation ID**: `1588b2ac37bd96e4113b8a7d1afbb9c3`
**Transaction Timeline**:
| Event | Duration |
|-------|----------|
| AI unknown - Processing request - Instructions | 250.4 µs |
| AI unknown - Processing request - User message | 144.4 µs |
| AI unknown - Processing request - User message | 149.6 µs |
| mcp_kb_knowledgebase842_8a5nl.knowledge_base_retrieve execute_tool | 98.4 ms |
| AI unknown - Working - Generating response | 533.8 ms |
**Tool Message Content**: Empty or minimal
### Evidence of Missing Data in Final Response
The agent's final response indicates it could NOT retrieve the requested data:
```json
[{"role": "assistant", "parts": [{"type": "text", "content": "I couldn't retrieve the list of all tests from the catalog at this moment. Could you please specify if you want tests from a particular category or any other specific details? This will help me provide you with more accurate information."}]}]
```
This suggests:
1. The knowledge base tool was invoked (visible in trace)
2. The tool execution completed (98.4 ms duration)
3. However, the retrieved content (if any) is not visible in telemetry
4. The agent was unable to use the retrieved data, possibly due to approval flow issues
### Missing MCP Approval Request in Telemetry
The SDK correctly receives and processes `mcp_approval_request` events (as evidenced by the sample code handling them), but these events are **not captured in Application Insights telemetry**. This means:
- No visibility into which tools requested approval
- No audit trail of approval decisions
- Cannot debug approval-related failures
- Cannot measure approval latency impact
## Screenshots
## Impact
- **Debugging**: Cannot troubleshoot why knowledge retrieval may fail or return incomplete results
- **Auditing**: No record of what data was searched, retrieved, or what approvals were granted
- **Security/Compliance**: MCP tool approval decisions are not logged, creating compliance gaps
- **Cost Analysis**: Cannot correlate knowledge base usage with specific queries
- **Quality Improvement**: Cannot analyze retrieval patterns to improve prompts or knowledge base configuration
- **Approval Workflow Analysis**: Cannot measure or optimize the approval flow performance
## Related Issues
- #44638 - Similar telemetry gap for Azure AI Search tool
Contributor guide
Research direction
Start by reproducing the streaming Azure AI Foundry agent example and compare the SDK's response.output_item.done MCP events with the Application Insights End-to-End Transaction trace. Trace the telemetry emitted for the knowledge_base_retrieve execute_tool span and mcp_approval_request event. Done means tool inputs and outputs, approval requests and responses, and relevant knowledge-base metadata are visible in telemetry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100