Azure / Azure/azure-sdk-for-python

TaskAdherenceEvaluator silently drops system_message and tool_calls kwargs - no error, no warning

Open
#46,780 1 comment 0 reactions 0 assignees View on GitHub
bug Client customer-reported Evaluation needs-team-attention Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

**Package**: azure-ai-evaluation
**Version**: 1.16.6 (also reproducible in earlier versions)
**Python**: 3.10+

**Describe the bug**
TaskAdherenceEvaluator.__call__() accepts arbitrary **kwargs, so passing system_message and tool_calls as keyword arguments does not raise any error. However, internally these values are silently overwritten - the evaluator extracts system_message from the query conversation (looking for {"role": "system"}) and tool_calls from the response messages.

This means a caller who passes:
```
task_adherence_evaluator(
system_message="You are a helpful agent...",
query="What is the status of order #123?",
response="Your order has been shipped.",
tool_calls='[{"name": "get_order", "arguments": {"id": "123"}}]'
)
```
gets no error, but:

system_message → overwritten by " " (empty string, since query is a plain string with no role: system)
tool_calls → overwritten by " " (empty string, since response is a plain string with no tool call messages)

The evaluator then runs against an incomplete context, producing incorrect scores with no indication that inputs were dropped.

**To Reproduce**
```
import os
from azure.ai.evaluation import TaskAdherenceEvaluator

model_config = {
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_KEY"],
"azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
}

evaluator = TaskAdherenceEvaluator(model_config=model_config)

# Case 1: system_message as kwarg (SILENTLY DROPPED)
result_bad = evaluator(
system_message="You must always verify order status before responding.",
query="What is the status of order #123?",
response="Your order has been shipped.",
)

# Case 2: system_message embedded in query (WORKS CORRECTLY)
result_good = evaluator(
query=[
{"role": "system", "content": "You must always verify order status before responding."},
{"role": "user", "content": "What is the status of order #123?"},
],
response=[
{"role": "assistant", "content": "Your order has been shipped."},
],
)

# result_bad and result_good will differ —
# result_bad evaluates WITHOUT system instructions (silent data loss)
```

**Expected behavior**
Option A: TaskAdherenceEvaluator should accept system_message and tool_calls as explicit top-level parameters and use them directly - consistent with the internal prompty definition which already has:

```
inputs:
system_message: type: string
query: type: string
response: type: string
tool_calls: type: string
```
Option B: If system_message or tool_calls are passed as kwargs but the evaluator does not use them, it should raise a warning or error instead of silently dropping them.

**Actual behaviour**

1. __call__ accepts **kwargs → no TypeError
2. In _do_eval(), local variable system_message = " " overwrites any kwarg value
3. system_message is only populated from query messages with role == "system"
4. tool_calls is only populated from response messages
5. No warning logged, no error raised
6. Evaluator produces scores based on incomplete context

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.