Azure / Azure/azure-sdk-for-python

[BUG] azure-ai-evaluation: Cannot run built-in evaluators against multi-turn conversation input

Open
#45,077 5 comments 1 reaction 3 assignees Claimed by @mikhail View on GitHub
customer-reported Evaluation needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
2d 2h
Merged PRs (30d)
202

Description

- **Package Name**: azure-ai-evaluation
- **Package Version**: 1.15.0
- **Operating System**: Windows

**Describe the bug**
Error raised when running the following script.
Error: azure.ai.evaluation._exceptions.EvaluationException: (InternalError) unhashable type: 'list'

**To Reproduce**
data.csv:
```csv
conversation
"{""messages"": [{""role"": ""user"", ""content"": ""Hi""}, {""role"": ""assistant"", ""content"": ""Hello""}]}"
"{""messages"": [{""role"": ""user"", ""content"": ""Hi""}, {""role"": ""assistant"", ""content"": ""Hello""}, {""role"": ""user"", ""content"": ""How are you?""}, {""role"": ""assistant"", ""content"": ""I am fine""}]}"
```

main.py:
```py
import json
import os
from azure.ai.evaluation import (
evaluate,
CoherenceEvaluator,
AzureOpenAIModelConfiguration,
)
from azure.identity import DefaultAzureCredential

MODEL_CONFIG = AzureOpenAIModelConfiguration(
azure_endpoint="xxx",
azure_deployment="xxx",
api_version="2024-12-01-preview",
)

def target_function(conversation: str):
return {"conversation": json.loads(conversation)}

if __name__ == "__main__":
data_path = os.path.join(os.path.dirname(__file__), "data.csv")

result = evaluate(
data=data_path,
target=target_function,
evaluators={
"coherence": CoherenceEvaluator(
model_config=MODEL_CONFIG,
credential=DefaultAzureCredential(),
# is_reasoning_model=True,
)
},
evaluator_config={
"default": {
"column_mapping": {
"conversation": "${target.conversation}",
}
}
},
)
print(result)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.