Azure / Azure/azure-sdk-for-python
Azure AI Evaluation: Unable to track tool call and tool call accuracy for Azure AI Search
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
**Describe the bug**
I created an agent in Foundry with Azure AI Search as tool and tried evaluating that with similarity, tool call accuracy, and tool call success metrics. For tool call accuracy I am getting error: 100% of the batch run failed. 'str' object has no attribute 'get'. But I can see tool call in traces and for some queries I see 'Pass' for tool call success. Even though in one of the reasons for tool call success I can see tool call to azure ai search is getting mentioned, but evaluator is missing it and cannot validate it.
**To Reproduce**
Steps to reproduce the behavior:
1. Create an agent with Azure AI Search as tool.
2. Create evaluation for this agent with tool call accuracy as metric.
**Expected behavior**
Tool call gets recognized and validate if content was retrieved from index.
Contributor guide
Assessment
This issue has not been assessed yet.