Azure / Azure/azure-sdk-for-python

TaskAdherenceEvaluator silently drops system_message and tool_calls kwargs - no error, no warning

Offen
#46,780 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug Client customer-reported Evaluation needs-team-attention Service Attention
Vorherrschende Sprache
Python
Sterne
5.6k
Forks
3.4k
Ø Merge
2 T. 2 Std.
Gemergte PRs (30 T.)
213

Beschreibung

**Package**: azure-ai-evaluation
**Version**: 1.16.6 (also reproducible in earlier versions)
**Python**: 3.10+

**Describe the bug**
TaskAdherenceEvaluator.__call__() accepts arbitrary **kwargs, so passing system_message and tool_calls as keyword arguments does not raise any error. However, internally these values are silently overwritten - the evaluator extracts system_message from the query conversation (looking for {"role": "system"}) and tool_calls from the response messages.

This means a caller who passes:
```
task_adherence_evaluator(
system_message="You are a helpful agent...",
query="What is the status of order #123?",
response="Your order has been shipped.",
tool_calls='[{"name": "get_order", "arguments": {"id": "123"}}]'
)
```
gets no error, but:

system_message → overwritten by " " (empty string, since query is a plain string with no role: system)
tool_calls → overwritten by " " (empty string, since response is a plain string with no tool call messages)

The evaluator then runs against an incomplete context, producing incorrect scores with no indication that inputs were dropped.

**To Reproduce**
```
import os
from azure.ai.evaluation import TaskAdherenceEvaluator

model_config = {
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_KEY"],
"azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
}

evaluator = TaskAdherenceEvaluator(model_config=model_config)

# Case 1: system_message as kwarg (SILENTLY DROPPED)
result_bad = evaluator(
system_message="You must always verify order status before responding.",
query="What is the status of order #123?",
response="Your order has been shipped.",
)

# Case 2: system_message embedded in query (WORKS CORRECTLY)
result_good = evaluator(
query=[
{"role": "system", "content": "You must always verify order status before responding."},
{"role": "user", "content": "What is the status of order #123?"},
],
response=[
{"role": "assistant", "content": "Your order has been shipped."},
],
)

# result_bad and result_good will differ —
# result_bad evaluates WITHOUT system instructions (silent data loss)
```

**Expected behavior**
Option A: TaskAdherenceEvaluator should accept system_message and tool_calls as explicit top-level parameters and use them directly - consistent with the internal prompty definition which already has:

```
inputs:
system_message: type: string
query: type: string
response: type: string
tool_calls: type: string
```
Option B: If system_message or tool_calls are passed as kwargs but the evaluator does not use them, it should raise a warning or error instead of silently dropping them.

**Actual behaviour**

1. __call__ accepts **kwargs → no TypeError
2. In _do_eval(), local variable system_message = " " overwrites any kwarg value
3. system_message is only populated from query messages with role == "system"
4. tool_calls is only populated from response messages
5. No warning logged, no error raised
6. Evaluator produces scores based on incomplete context

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with TaskAdherenceEvaluator.__call__() and _do_eval() in the azure-ai-evaluation package, then compare their inputs with the internal prompty definition listing system_message, query, response, and tool_calls. Reproduce the keyword-argument cases from the issue and verify that the evaluator either uses those values explicitly or reports unsupported inputs instead of silently discarding them.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
azure, python
Bereich
ai
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
50/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.