Azure / Azure/azure-sdk-for-python
TaskAdherenceEvaluator silently drops system_message and tool_calls kwargs - no error, no warning
- Lingua principale
- Python
- Stelle
- 5.6k
- Fork
- 3.4k
- Merge medio
- 2g
- PR unite (30g)
- 217
Descrizione
**Package**: azure-ai-evaluation
**Version**: 1.16.6 (also reproducible in earlier versions)
**Python**: 3.10+
**Describe the bug**
TaskAdherenceEvaluator.__call__() accepts arbitrary **kwargs, so passing system_message and tool_calls as keyword arguments does not raise any error. However, internally these values are silently overwritten - the evaluator extracts system_message from the query conversation (looking for {"role": "system"}) and tool_calls from the response messages.
This means a caller who passes:
```
task_adherence_evaluator(
system_message="You are a helpful agent...",
query="What is the status of order #123?",
response="Your order has been shipped.",
tool_calls='[{"name": "get_order", "arguments": {"id": "123"}}]'
)
```
gets no error, but:
system_message → overwritten by " " (empty string, since query is a plain string with no role: system)
tool_calls → overwritten by " " (empty string, since response is a plain string with no tool call messages)
The evaluator then runs against an incomplete context, producing incorrect scores with no indication that inputs were dropped.
**To Reproduce**
```
import os
from azure.ai.evaluation import TaskAdherenceEvaluator
model_config = {
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_KEY"],
"azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
}
evaluator = TaskAdherenceEvaluator(model_config=model_config)
# Case 1: system_message as kwarg (SILENTLY DROPPED)
result_bad = evaluator(
system_message="You must always verify order status before responding.",
query="What is the status of order #123?",
response="Your order has been shipped.",
)
# Case 2: system_message embedded in query (WORKS CORRECTLY)
result_good = evaluator(
query=[
{"role": "system", "content": "You must always verify order status before responding."},
{"role": "user", "content": "What is the status of order #123?"},
],
response=[
{"role": "assistant", "content": "Your order has been shipped."},
],
)
# result_bad and result_good will differ —
# result_bad evaluates WITHOUT system instructions (silent data loss)
```
**Expected behavior**
Option A: TaskAdherenceEvaluator should accept system_message and tool_calls as explicit top-level parameters and use them directly - consistent with the internal prompty definition which already has:
```
inputs:
system_message: type: string
query: type: string
response: type: string
tool_calls: type: string
```
Option B: If system_message or tool_calls are passed as kwargs but the evaluator does not use them, it should raise a warning or error instead of silently dropping them.
**Actual behaviour**
1. __call__ accepts **kwargs → no TypeError
2. In _do_eval(), local variable system_message = " " overwrites any kwarg value
3. system_message is only populated from query messages with role == "system"
4. tool_calls is only populated from response messages
5. No warning logged, no error raised
6. Evaluator produces scores based on incomplete context
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with TaskAdherenceEvaluator.__call__() and _do_eval() in the azure-ai-evaluation package, then compare their inputs with the internal prompty definition listing system_message, query, response, and tool_calls. Reproduce the keyword-argument cases from the issue and verify that the evaluator either uses those values explicitly or reports unsupported inputs instead of silently discarding them.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- azure, python
- Ambito
- ai
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 50/100