Azure / Azure/azure-sdk-for-python
azure.ai.evaluation SDK cannot handle `conversation` input from a csv file.
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- **Package Name**: azure.ai.evaluation
- **Package Version**: 1.8.0
- **Operating System**: Windows 11
- **Python Version**: 3.12.9
**Describe the bug**
In a csv file,
**To Reproduce**
Steps to reproduce the behavior:
pip install azure.ai.evaluation==1.8.0
Setup file `data.csv`:
```
conversation
"{""messages"": [{""role"": ""user"", ""content"": ""Hello""}, {""role"": ""assistant"", ""content"": ""Hello""}]}"
````
Setup file main.py
```python
from pathlib import Path
from azure.ai.evaluation import (
evaluate,
CoherenceEvaluator,
AzureOpenAIModelConfiguration,
)
dataset_path = Path("data.csv").resolve()
model_config = AzureOpenAIModelConfiguration(
azure_deployment="xxx",
azure_endpoint="xxx",
)
ret = evaluate(
data=dataset_path,
evaluators={"coherence": CoherenceEvaluator(model_config=model_config)},
evaluator_config={"default": {"conversation": "${data.conversation}"}},
)
```
**Expected behavior**
It should output coherence score
**Additional context**
In line 346 of the code https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py
Looks like the input conversation is a string, instead of a Dict, the previous check `not in` doesn't take effect, because the stringified conversation json does contain the substring 'message'.
Contributor guide
Assessment
This issue has not been assessed yet.