Azure / Azure/azure-sdk-for-python
Community package: azure-ai-evaluation-openeval-adapter (evaluate() data/results <-> EvalPort interchange)
- Vorherrschende Sprache
- Python
- Sterne
- 5.6k
- Forks
- 3.4k
- Ø Merge
- 2 T.
- Gemergte PRs (30 T.)
- 217
Beschreibung
Hi Azure AI Evaluation team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open, schema-validated JSON interchange format for portable LLM evaluation test cases, graders, suites, and results. It already has independently-tested adapter packages for ~20 other eval/observability frameworks (MLflow, LangSmith, Ragas, Vertex AI Gen AI Evaluation, Hugging Face `evaluate`, and others), and I built one for `azure-ai-evaluation` the same way.
This isn't a request for a change in this repo — I'm not proposing new API surface or asking for a design review, just flagging a working, tested community package in case it's useful to know about or link from docs.
**[`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter)**
```python
from azure.ai.evaluation import F1ScoreEvaluator, evaluate
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set
suite = to_openeval(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()}, suite_id="my_eval_suite")
assert validate_suite(suite).valid
result = evaluate(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()})
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```
`to_openeval()` accepts exactly what `evaluate()` itself accepts for `data`/`evaluators`, so it's a pure format bridge rather than new infrastructure. The one design choice worth flagging: every evaluator (local NLP metrics like F1/BLEU/ROUGE, AI-assisted evaluators needing a live `model_config`, and the content-safety evaluators needing a live Foundry project) maps to EvalPort's `custom` grader type rather than being force-fit into `semantic_similarity` or `llm_judge` — those types require params (`threshold`, `prompt`) this adapter can't honestly fabricate from the outside. Full mapping table and the flat-row parsing logic (recovering per-metric score/passed/reason from `evaluate()`'s real `outputs..*` column convention) are in the [README](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter#readme).
21 tests, all passing locally against the real installed `azure-ai-evaluation` package and EvalPort's real `validate_suite()`/`validate_result_set()` — not mocked.
No action needed — this lives entirely outside `azure-sdk-for-python` as an independent package (`pip install` via `git+`, not yet on PyPI). Flagging mainly for discoverability; happy to adjust the mapping if the evaluation module's public API shifts, or to send a one-line docs PR if there's a community-packages list this belongs on.
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Beitragsleitfaden
Rechercherichtung
Beginne mit der verlinkten azure-ai-evaluation-openeval-adapter README und prüfe, ob dieses Repository eine community-packages-Liste oder einen anderen Einstiegspunkt zur Dokumentation hat. Es wird keine Änderung am Repository angefordert; falls eine passende Auflistung existiert, wäre done ein prägnanter Link zur besseren Auffindbarkeit, andernfalls muss das issue geklärt werden.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- azure, python
- Bereich
- documentation
- Issue-Typ
- Dokumentation
- Schwierigkeit
- 1/5
- Geschätzter Aufwand
- Unter einer Stunde
- Aktivitätsstatus
- Aktiv
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 35/100