Azure / Azure/azure-sdk-for-python
[BUG] azure-ai-evaluation: Cannot run built-in evaluators against multi-turn conversation input
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 202
Description
- **Package Name**: azure-ai-evaluation
- **Package Version**: 1.15.0
- **Operating System**: Windows
**Describe the bug**
Error raised when running the following script.
Error: azure.ai.evaluation._exceptions.EvaluationException: (InternalError) unhashable type: 'list'
**To Reproduce**
data.csv:
```csv
conversation
"{""messages"": [{""role"": ""user"", ""content"": ""Hi""}, {""role"": ""assistant"", ""content"": ""Hello""}]}"
"{""messages"": [{""role"": ""user"", ""content"": ""Hi""}, {""role"": ""assistant"", ""content"": ""Hello""}, {""role"": ""user"", ""content"": ""How are you?""}, {""role"": ""assistant"", ""content"": ""I am fine""}]}"
```
main.py:
```py
import json
import os
from azure.ai.evaluation import (
evaluate,
CoherenceEvaluator,
AzureOpenAIModelConfiguration,
)
from azure.identity import DefaultAzureCredential
MODEL_CONFIG = AzureOpenAIModelConfiguration(
azure_endpoint="xxx",
azure_deployment="xxx",
api_version="2024-12-01-preview",
)
def target_function(conversation: str):
return {"conversation": json.loads(conversation)}
if __name__ == "__main__":
data_path = os.path.join(os.path.dirname(__file__), "data.csv")
result = evaluate(
data=data_path,
target=target_function,
evaluators={
"coherence": CoherenceEvaluator(
model_config=MODEL_CONFIG,
credential=DefaultAzureCredential(),
# is_reasoning_model=True,
)
},
evaluator_config={
"default": {
"column_mapping": {
"conversation": "${target.conversation}",
}
}
},
)
print(result)
```
Contributor guide
Assessment
This issue has not been assessed yet.