Azure / Azure/azure-sdk-for-python

azure-ai-evaluation: Improper handling of missing fields by evaluators during batch evaluation

Offen
#42,358 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
customer-reported Evaluation feature-request needs-team-attention
Vorherrschende Sprache
Python
Sterne
5.6k
Forks
3.4k
Ø Merge
2 T. 2 Std.
Gemergte PRs (30 T.)
213

Beschreibung

- **Package Name**: `azure-ai-evaluation`
- **Package Version**: 1.8.0
- **Operating System**: macOS Sequoia 15.5
- **Python Version**: 3.11.8

**Describe the bug**
There is a bug I've found which I reported details of [here](https://github.com/orgs/azure-ai-foundry/discussions/111). The gist is that, when you provide Q&A sets where ground truth is present for some, but not all, the evaluation results will be heavily skewed for response completeness given all Q&A sets without ground truth will receive a response completeness score of 1. In addition, this scenario also elicits errors in the logs during the batch evaluation step.

**To Reproduce**
Steps to reproduce the behavior:
1. Create an evaluation-compatible Q&A sets file with a list of questions and their corresponding answers, some of which (but not all) having `ground_truth`, too.
2. Run batch evaluations on that file, requesting both response completeness and any other evaluators.

**Expected behavior**
My expectation for this scenario would be for results to not be skewed by Q&A sets where the ground truth was not provided. Instead, the response completeness evaluation should be avoided altogether for those specific Q&A sets. In other words, this is the breakdown of the possible scenarios:
1. Q&A set with all required fields: Run all provided evaluators.
2. Q&A set without ground truth: Run all evaluators **except** response completeness.

Importantly, this doesn't apply to just response completeness, but more broadly, any scenario where you may have missing fields for certain evaluators. There should be a preventative step that looks at the type of data of the given Q&A set and determines what evaluators qualify.

**Screenshots**
N/A

**Additional context**
- See my initial discussion post on this issue [here](https://github.com/orgs/azure-ai-foundry/discussions/111) where I talked with Nitya.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne im azure-ai-evaluation-Batch-Evaluierungsablauf und reproduziere einen gemischten Q&A-Satz mit und ohne ground_truth, während du die Antwortvollständigkeit anforderst. Verfolge, wie die von den Evaluatoren benötigten Felder überprüft werden; abgeschlossen ist die Arbeit, wenn Evaluatoren für nicht zulässige Q&A-Sätze übersprungen werden, die Scores nicht verzerrt sind und die protokollierten Fehler vermieden werden.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
azure, python
Bereich
ai, machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.