Azure / Azure/azure-sdk-for-python

Bug in QAEvaluator Evaluation Object

Open
#43,149 1 comment 0 reactions 0 assignees View on GitHub
AI Client customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- azure-ai-evaluation:
- 1.11.0:
- MacOS:
- Python 3.11.9:

When using the QAEvaluator object the F1 threshold is defaulting to 3 when it should be 0.5. Not only that, it is passing below the threshold when only above the threshold should be passing

Run the QAEvaluator object on a simple example and it should come up.

The default threshold for F1 should be 0.5, and it should pass if the value is above the threshold (not below).

Image

Image

I tried to run this with setting the default in the object to 0.5 manually and it still defaulted to 3.0.

And Secondly, if your agent outputs happen to be just a string composed of numbers reading in from the jsonl file converts the column to integer and QAEvaluator will fail at a ".lower()" step.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.