Azure / Azure/azure-sdk-for-python

Azure AI Evaluation: Unable to track tool call and tool call accuracy for Azure AI Search

Offen
#44,480 2 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @w-javed Auf GitHub ansehen
bug customer-reported Evaluation needs-team-attention question Service Attention
Vorherrschende Sprache
Python
Sterne
5.6k
Forks
3.4k
Ø Merge
2 T.
Gemergte PRs (30 T.)
217

Beschreibung

**Describe the bug**
I created an agent in Foundry with Azure AI Search as tool and tried evaluating that with similarity, tool call accuracy, and tool call success metrics. For tool call accuracy I am getting error: 100% of the batch run failed. 'str' object has no attribute 'get'. But I can see tool call in traces and for some queries I see 'Pass' for tool call success. Even though in one of the reasons for tool call success I can see tool call to azure ai search is getting mentioned, but evaluator is missing it and cannot validate it.

Image

**To Reproduce**
Steps to reproduce the behavior:
1. Create an agent with Azure AI Search as tool.
2. Create evaluation for this agent with tool call accuracy as metric.

**Expected behavior**
Tool call gets recognized and validate if content was retrieved from index.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.