Azure / Azure/azure-sdk-for-python
Community package: azure-ai-evaluation-openeval-adapter (evaluate() data/results <-> EvalPort interchange)
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
Hi Azure AI Evaluation team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open, schema-validated JSON interchange format for portable LLM evaluation test cases, graders, suites, and results. It already has independently-tested adapter packages for ~20 other eval/observability frameworks (MLflow, LangSmith, Ragas, Vertex AI Gen AI Evaluation, Hugging Face `evaluate`, and others), and I built one for `azure-ai-evaluation` the same way.
This isn't a request for a change in this repo — I'm not proposing new API surface or asking for a design review, just flagging a working, tested community package in case it's useful to know about or link from docs.
**[`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter)**
```python
from azure.ai.evaluation import F1ScoreEvaluator, evaluate
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set
suite = to_openeval(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()}, suite_id="my_eval_suite")
assert validate_suite(suite).valid
result = evaluate(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()})
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```
`to_openeval()` accepts exactly what `evaluate()` itself accepts for `data`/`evaluators`, so it's a pure format bridge rather than new infrastructure. The one design choice worth flagging: every evaluator (local NLP metrics like F1/BLEU/ROUGE, AI-assisted evaluators needing a live `model_config`, and the content-safety evaluators needing a live Foundry project) maps to EvalPort's `custom` grader type rather than being force-fit into `semantic_similarity` or `llm_judge` — those types require params (`threshold`, `prompt`) this adapter can't honestly fabricate from the outside. Full mapping table and the flat-row parsing logic (recovering per-metric score/passed/reason from `evaluate()`'s real `outputs..*` column convention) are in the [README](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter#readme).
21 tests, all passing locally against the real installed `azure-ai-evaluation` package and EvalPort's real `validate_suite()`/`validate_result_set()` — not mocked.
No action needed — this lives entirely outside `azure-sdk-for-python` as an independent package (`pip install` via `git+`, not yet on PyPI). Flagging mainly for discoverability; happy to adjust the mapping if the evaluation module's public API shifts, or to send a one-line docs PR if there's a community-packages list this belongs on.
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Contributor guide
Research direction
Start with the linked azure-ai-evaluation-openeval-adapter README and check whether this repository has a community-packages list or other documentation entry point. No repository change is requested; if an appropriate listing exists, done would be a concise discoverability link, otherwise the issue needs clarification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100