Azure / Azure/azure-sdk-for-python

azure.ai.evaluation SDK cannot handle `conversation` input from a csv file.

Open
#41,642 3 comments 1 reaction 2 assignees Claimed by @singankit View on GitHub
Client customer-reported Evaluation feature-request needs-team-attention Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: azure.ai.evaluation
- **Package Version**: 1.8.0
- **Operating System**: Windows 11
- **Python Version**: 3.12.9

**Describe the bug**
In a csv file,

**To Reproduce**
Steps to reproduce the behavior:

pip install azure.ai.evaluation==1.8.0
Setup file `data.csv`:
```
conversation
"{""messages"": [{""role"": ""user"", ""content"": ""Hello""}, {""role"": ""assistant"", ""content"": ""Hello""}]}"
````
Setup file main.py
```python
from pathlib import Path
from azure.ai.evaluation import (
evaluate,
CoherenceEvaluator,
AzureOpenAIModelConfiguration,
)

dataset_path = Path("data.csv").resolve()

model_config = AzureOpenAIModelConfiguration(
azure_deployment="xxx",
azure_endpoint="xxx",
)

ret = evaluate(
data=dataset_path,
evaluators={"coherence": CoherenceEvaluator(model_config=model_config)},
evaluator_config={"default": {"conversation": "${data.conversation}"}},
)

```

**Expected behavior**
It should output coherence score

**Additional context**
In line 346 of the code https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py

Looks like the input conversation is a string, instead of a Dict, the previous check `not in` doesn't take effect, because the stringified conversation json does contain the substring 'message'.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.