Azure / Azure/azure-sdk-for-python

Community package: azure-ai-evaluation-openeval-adapter (evaluate() data/results <-> EvalPort interchange)

Offen
#48,971 0 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
Evaluation feature-request
Vorherrschende Sprache
Python
Sterne
5.6k
Forks
3.4k
Ø Merge
2 T.
Gemergte PRs (30 T.)
217

Beschreibung

Hi Azure AI Evaluation team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open, schema-validated JSON interchange format for portable LLM evaluation test cases, graders, suites, and results. It already has independently-tested adapter packages for ~20 other eval/observability frameworks (MLflow, LangSmith, Ragas, Vertex AI Gen AI Evaluation, Hugging Face `evaluate`, and others), and I built one for `azure-ai-evaluation` the same way.

This isn't a request for a change in this repo — I'm not proposing new API surface or asking for a design review, just flagging a working, tested community package in case it's useful to know about or link from docs.

**[`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter)**

```python
from azure.ai.evaluation import F1ScoreEvaluator, evaluate
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set

suite = to_openeval(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()}, suite_id="my_eval_suite")
assert validate_suite(suite).valid

result = evaluate(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()})
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```

`to_openeval()` accepts exactly what `evaluate()` itself accepts for `data`/`evaluators`, so it's a pure format bridge rather than new infrastructure. The one design choice worth flagging: every evaluator (local NLP metrics like F1/BLEU/ROUGE, AI-assisted evaluators needing a live `model_config`, and the content-safety evaluators needing a live Foundry project) maps to EvalPort's `custom` grader type rather than being force-fit into `semantic_similarity` or `llm_judge` — those types require params (`threshold`, `prompt`) this adapter can't honestly fabricate from the outside. Full mapping table and the flat-row parsing logic (recovering per-metric score/passed/reason from `evaluate()`'s real `outputs..*` column convention) are in the [README](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter#readme).

21 tests, all passing locally against the real installed `azure-ai-evaluation` package and EvalPort's real `validate_suite()`/`validate_result_set()` — not mocked.

No action needed — this lives entirely outside `azure-sdk-for-python` as an independent package (`pip install` via `git+`, not yet on PyPI). Flagging mainly for discoverability; happy to adjust the mapping if the evaluation module's public API shifts, or to send a one-line docs PR if there's a community-packages list this belongs on.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit der verlinkten azure-ai-evaluation-openeval-adapter README und prüfe, ob dieses Repository eine community-packages-Liste oder einen anderen Einstiegspunkt zur Dokumentation hat. Es wird keine Änderung am Repository angefordert; falls eine passende Auflistung existiert, wäre done ein prägnanter Link zur besseren Auffindbarkeit, andernfalls muss das issue geklärt werden.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
azure, python
Bereich
documentation
Issue-Typ
Dokumentation
Schwierigkeit
1/5
Geschätzter Aufwand
Unter einer Stunde
Aktivitätsstatus
Aktiv
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.