Azure / Azure/azure-sdk-for-python

Azure AI Evaluation: Unable to track tool call and tool call accuracy for Azure AI Search

Open
#44,480 2 comments 0 reactions 1 assignee Claimed by @w-javed View on GitHub
bug customer-reported Evaluation needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

**Describe the bug**
I created an agent in Foundry with Azure AI Search as tool and tried evaluating that with similarity, tool call accuracy, and tool call success metrics. For tool call accuracy I am getting error: 100% of the batch run failed. 'str' object has no attribute 'get'. But I can see tool call in traces and for some queries I see 'Pass' for tool call success. Even though in one of the reasons for tool call success I can see tool call to azure ai search is getting mentioned, but evaluator is missing it and cannot validate it.

Image

**To Reproduce**
Steps to reproduce the behavior:
1. Create an agent with Azure AI Search as tool.
2. Create evaluation for this agent with tool call accuracy as metric.

**Expected behavior**
Tool call gets recognized and validate if content was retrieved from index.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.