Azure / Azure/azure-sdk-for-python

TaskAdherenceEvaluator silently drops system_message and tool_calls kwargs - no error, no warning

Aperta
#46,780 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug Client customer-reported Evaluation needs-team-attention Service Attention
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g
PR unite (30g)
217

Descrizione

**Package**: azure-ai-evaluation
**Version**: 1.16.6 (also reproducible in earlier versions)
**Python**: 3.10+

**Describe the bug**
TaskAdherenceEvaluator.__call__() accepts arbitrary **kwargs, so passing system_message and tool_calls as keyword arguments does not raise any error. However, internally these values are silently overwritten - the evaluator extracts system_message from the query conversation (looking for {"role": "system"}) and tool_calls from the response messages.

This means a caller who passes:
```
task_adherence_evaluator(
system_message="You are a helpful agent...",
query="What is the status of order #123?",
response="Your order has been shipped.",
tool_calls='[{"name": "get_order", "arguments": {"id": "123"}}]'
)
```
gets no error, but:

system_message → overwritten by " " (empty string, since query is a plain string with no role: system)
tool_calls → overwritten by " " (empty string, since response is a plain string with no tool call messages)

The evaluator then runs against an incomplete context, producing incorrect scores with no indication that inputs were dropped.

**To Reproduce**
```
import os
from azure.ai.evaluation import TaskAdherenceEvaluator

model_config = {
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_KEY"],
"azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
}

evaluator = TaskAdherenceEvaluator(model_config=model_config)

# Case 1: system_message as kwarg (SILENTLY DROPPED)
result_bad = evaluator(
system_message="You must always verify order status before responding.",
query="What is the status of order #123?",
response="Your order has been shipped.",
)

# Case 2: system_message embedded in query (WORKS CORRECTLY)
result_good = evaluator(
query=[
{"role": "system", "content": "You must always verify order status before responding."},
{"role": "user", "content": "What is the status of order #123?"},
],
response=[
{"role": "assistant", "content": "Your order has been shipped."},
],
)

# result_bad and result_good will differ —
# result_bad evaluates WITHOUT system instructions (silent data loss)
```

**Expected behavior**
Option A: TaskAdherenceEvaluator should accept system_message and tool_calls as explicit top-level parameters and use them directly - consistent with the internal prompty definition which already has:

```
inputs:
system_message: type: string
query: type: string
response: type: string
tool_calls: type: string
```
Option B: If system_message or tool_calls are passed as kwargs but the evaluator does not use them, it should raise a warning or error instead of silently dropping them.

**Actual behaviour**

1. __call__ accepts **kwargs → no TypeError
2. In _do_eval(), local variable system_message = " " overwrites any kwarg value
3. system_message is only populated from query messages with role == "system"
4. tool_calls is only populated from response messages
5. No warning logged, no error raised
6. Evaluator produces scores based on incomplete context

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with TaskAdherenceEvaluator.__call__() and _do_eval() in the azure-ai-evaluation package, then compare their inputs with the internal prompty definition listing system_message, query, response, and tool_calls. Reproduce the keyword-argument cases from the issue and verify that the evaluator either uses those values explicitly or reports unsupported inputs instead of silently discarding them.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
azure, python
Ambito
ai
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
50/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.