Azure / Azure/azure-sdk-for-python

Azure AI Evaluation: Unable to track tool call and tool call accuracy for Azure AI Search

Aperta
#44,480 2 commenti 0 reazioni 1 assegnatario Rivendicata da @w-javed Vedi su GitHub
bug customer-reported Evaluation needs-team-attention question Service Attention
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g
PR unite (30g)
217

Descrizione

**Describe the bug**
I created an agent in Foundry with Azure AI Search as tool and tried evaluating that with similarity, tool call accuracy, and tool call success metrics. For tool call accuracy I am getting error: 100% of the batch run failed. 'str' object has no attribute 'get'. But I can see tool call in traces and for some queries I see 'Pass' for tool call success. Even though in one of the reasons for tool call success I can see tool call to azure ai search is getting mentioned, but evaluator is missing it and cannot validate it.

Image

**To Reproduce**
Steps to reproduce the behavior:
1. Create an agent with Azure AI Search as tool.
2. Create evaluation for this agent with tool call accuracy as metric.

**Expected behavior**
Tool call gets recognized and validate if content was retrieved from index.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.