Azure / Azure/azure-sdk-for-python
TaskAdherenceEvaluator silently drops system_message and tool_calls kwargs - no error, no warning
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
**Package**: azure-ai-evaluation
**Version**: 1.16.6 (also reproducible in earlier versions)
**Python**: 3.10+
**Describe the bug**
TaskAdherenceEvaluator.__call__() accepts arbitrary **kwargs, so passing system_message and tool_calls as keyword arguments does not raise any error. However, internally these values are silently overwritten - the evaluator extracts system_message from the query conversation (looking for {"role": "system"}) and tool_calls from the response messages.
This means a caller who passes:
```
task_adherence_evaluator(
system_message="You are a helpful agent...",
query="What is the status of order #123?",
response="Your order has been shipped.",
tool_calls='[{"name": "get_order", "arguments": {"id": "123"}}]'
)
```
gets no error, but:
system_message → overwritten by " " (empty string, since query is a plain string with no role: system)
tool_calls → overwritten by " " (empty string, since response is a plain string with no tool call messages)
The evaluator then runs against an incomplete context, producing incorrect scores with no indication that inputs were dropped.
**To Reproduce**
```
import os
from azure.ai.evaluation import TaskAdherenceEvaluator
model_config = {
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_KEY"],
"azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
}
evaluator = TaskAdherenceEvaluator(model_config=model_config)
# Case 1: system_message as kwarg (SILENTLY DROPPED)
result_bad = evaluator(
system_message="You must always verify order status before responding.",
query="What is the status of order #123?",
response="Your order has been shipped.",
)
# Case 2: system_message embedded in query (WORKS CORRECTLY)
result_good = evaluator(
query=[
{"role": "system", "content": "You must always verify order status before responding."},
{"role": "user", "content": "What is the status of order #123?"},
],
response=[
{"role": "assistant", "content": "Your order has been shipped."},
],
)
# result_bad and result_good will differ —
# result_bad evaluates WITHOUT system instructions (silent data loss)
```
**Expected behavior**
Option A: TaskAdherenceEvaluator should accept system_message and tool_calls as explicit top-level parameters and use them directly - consistent with the internal prompty definition which already has:
```
inputs:
system_message: type: string
query: type: string
response: type: string
tool_calls: type: string
```
Option B: If system_message or tool_calls are passed as kwargs but the evaluator does not use them, it should raise a warning or error instead of silently dropping them.
**Actual behaviour**
1. __call__ accepts **kwargs → no TypeError
2. In _do_eval(), local variable system_message = " " overwrites any kwarg value
3. system_message is only populated from query messages with role == "system"
4. tool_calls is only populated from response messages
5. No warning logged, no error raised
6. Evaluator produces scores based on incomplete context
Contributor guide
Assessment
This issue has not been assessed yet.