Azure-Samples / Azure-Samples/ai-rag-chat-evaluator
Getting nan values in results for metrics
- Dominant language
- Python
- Stars
- 335
- Forks
- 113
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 1
Description
> Please provide us with the following information:
> ---------------------------------------------------------------
While running the py snippet i am getting null values for some metrices. Using below mentioned input.
question="What is the capital of France?",
context="France is in Europe and it's capital is Paris.",
answer="Paris is the capital of France."
truth = "Paris"
results = evaluate(
target=wrap_target,
data=testdata,
task_type="qa",
metrics_list=["gpt_groundedness","gpt_relevance","gpt_coherence","gpt_fluency","gpt_similarity", "hate_unfairness", "sexual", "violence", "self_harm"],
model_config= model_config
data_mapping={
"question": "question",
"context": "context",
"answer": "answer",
},
tracking=False,
output_path="./")
### This issue is for a: (mark with an `x`)
```
- [ print(results.metrics_summary )
- {'gpt_coherence': nan,
'gpt_similarity': nan,
'gpt_fluency': nan,
'gpt_relevance': 5.0,
'gpt_groundedness': 5.0,
'self_harm_defect_rate': nan,
'violence_defect_rate': nan,
'hate_unfairness_defect_rate': nan,
'sexual_defect_rate': nan} ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```
### Minimal steps to reproduce
>
### Any log messages given by the failure
> Fail writing properties '{'_azureml.evaluation_run': 'azure-ai-generative-parent'}' to run history: 'FileStore' object has no attribute 'get_host_creds'
### Expected/desired behavior
> I am getting the answer for gpt_groundedness and gpt_relevance but for other metrics getting null values.
### OS and Version?
> Windows 11
### Versions
>azure-ai-generative[evaluate]==1.0.0b8
promptflow==1.6.0
### Mention any other details that might be useful
> ---------------------------------------------------------------
> Thanks! We'll be in touch soon.
Contributor guide
Research direction
Start by reproducing the provided evaluate snippet with azure-ai-generative[evaluate]==1.0.0b8 and promptflow==1.6.0 on Windows 11, then inspect the metric outputs and the FileStore get_host_creds error. Done means determining why the listed metrics return nan while groundedness and relevance return values, or documenting the required compatible configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100