Azure-Samples / Azure-Samples/ai-rag-chat-evaluator

Getting nan values in results for metrics

Open
#80 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
335
Forks
113
Avg merge
3d 15h
Merged PRs (30d)
1

Description

> Please provide us with the following information:
> ---------------------------------------------------------------
While running the py snippet i am getting null values for some metrices. Using below mentioned input.

question="What is the capital of France?",
context="France is in Europe and it's capital is Paris.",
answer="Paris is the capital of France."
truth = "Paris"

results = evaluate(
target=wrap_target,
data=testdata,
task_type="qa",
metrics_list=["gpt_groundedness","gpt_relevance","gpt_coherence","gpt_fluency","gpt_similarity", "hate_unfairness", "sexual", "violence", "self_harm"],
model_config= model_config
data_mapping={
"question": "question",
"context": "context",
"answer": "answer",
},
tracking=False,
output_path="./")
### This issue is for a: (mark with an `x`)
```
- [ print(results.metrics_summary )
- {'gpt_coherence': nan,
'gpt_similarity': nan,
'gpt_fluency': nan,
'gpt_relevance': 5.0,
'gpt_groundedness': 5.0,
'self_harm_defect_rate': nan,
'violence_defect_rate': nan,
'hate_unfairness_defect_rate': nan,
'sexual_defect_rate': nan} ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```

### Minimal steps to reproduce
>

### Any log messages given by the failure
> Fail writing properties '{'_azureml.evaluation_run': 'azure-ai-generative-parent'}' to run history: 'FileStore' object has no attribute 'get_host_creds'

### Expected/desired behavior
> I am getting the answer for gpt_groundedness and gpt_relevance but for other metrics getting null values.

### OS and Version?
> Windows 11

### Versions
>azure-ai-generative[evaluate]==1.0.0b8
promptflow==1.6.0

### Mention any other details that might be useful

> ---------------------------------------------------------------
> Thanks! We'll be in touch soon.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided evaluate snippet with azure-ai-generative[evaluate]==1.0.0b8 and promptflow==1.6.0 on Windows 11, then inspect the metric outputs and the FileStore get_host_creds error. Done means determining why the listed metrics return nan while groundedness and relevance return values, or documenting the required compatible configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.