Azure / Azure/azure-sdk-for-python
TaskAdherenceEvaluator silently drops system_message and tool_calls kwargs - no error, no warning
- Vorherrschende Sprache
- Python
- Sterne
- 5.6k
- Forks
- 3.4k
- Ø Merge
- 2 T. 2 Std.
- Gemergte PRs (30 T.)
- 213
Beschreibung
**Package**: azure-ai-evaluation
**Version**: 1.16.6 (also reproducible in earlier versions)
**Python**: 3.10+
**Describe the bug**
TaskAdherenceEvaluator.__call__() accepts arbitrary **kwargs, so passing system_message and tool_calls as keyword arguments does not raise any error. However, internally these values are silently overwritten - the evaluator extracts system_message from the query conversation (looking for {"role": "system"}) and tool_calls from the response messages.
This means a caller who passes:
```
task_adherence_evaluator(
system_message="You are a helpful agent...",
query="What is the status of order #123?",
response="Your order has been shipped.",
tool_calls='[{"name": "get_order", "arguments": {"id": "123"}}]'
)
```
gets no error, but:
system_message → overwritten by " " (empty string, since query is a plain string with no role: system)
tool_calls → overwritten by " " (empty string, since response is a plain string with no tool call messages)
The evaluator then runs against an incomplete context, producing incorrect scores with no indication that inputs were dropped.
**To Reproduce**
```
import os
from azure.ai.evaluation import TaskAdherenceEvaluator
model_config = {
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_KEY"],
"azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
}
evaluator = TaskAdherenceEvaluator(model_config=model_config)
# Case 1: system_message as kwarg (SILENTLY DROPPED)
result_bad = evaluator(
system_message="You must always verify order status before responding.",
query="What is the status of order #123?",
response="Your order has been shipped.",
)
# Case 2: system_message embedded in query (WORKS CORRECTLY)
result_good = evaluator(
query=[
{"role": "system", "content": "You must always verify order status before responding."},
{"role": "user", "content": "What is the status of order #123?"},
],
response=[
{"role": "assistant", "content": "Your order has been shipped."},
],
)
# result_bad and result_good will differ —
# result_bad evaluates WITHOUT system instructions (silent data loss)
```
**Expected behavior**
Option A: TaskAdherenceEvaluator should accept system_message and tool_calls as explicit top-level parameters and use them directly - consistent with the internal prompty definition which already has:
```
inputs:
system_message: type: string
query: type: string
response: type: string
tool_calls: type: string
```
Option B: If system_message or tool_calls are passed as kwargs but the evaluator does not use them, it should raise a warning or error instead of silently dropping them.
**Actual behaviour**
1. __call__ accepts **kwargs → no TypeError
2. In _do_eval(), local variable system_message = " " overwrites any kwarg value
3. system_message is only populated from query messages with role == "system"
4. tool_calls is only populated from response messages
5. No warning logged, no error raised
6. Evaluator produces scores based on incomplete context
Beitragsleitfaden
Rechercherichtung
Start with TaskAdherenceEvaluator.__call__() and _do_eval() in the azure-ai-evaluation package, then compare their inputs with the internal prompty definition listing system_message, query, response, and tool_calls. Reproduce the keyword-argument cases from the issue and verify that the evaluator either uses those values explicitly or reports unsupported inputs instead of silently discarding them.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- azure, python
- Bereich
- ai
- Issue-Typ
- Bug
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 50/100