Azure / Azure/azure-sdk-for-python
Bug in QAEvaluator Evaluation Object
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- azure-ai-evaluation:
- 1.11.0:
- MacOS:
- Python 3.11.9:
When using the QAEvaluator object the F1 threshold is defaulting to 3 when it should be 0.5. Not only that, it is passing below the threshold when only above the threshold should be passing
Run the QAEvaluator object on a simple example and it should come up.
The default threshold for F1 should be 0.5, and it should pass if the value is above the threshold (not below).
I tried to run this with setting the default in the object to 0.5 manually and it still defaulted to 3.0.
And Secondly, if your agent outputs happen to be just a string composed of numbers reading in from the jsonl file converts the column to integer and QAEvaluator will fail at a ".lower()" step.
Contributor guide
Assessment
This issue has not been assessed yet.