Azure / Azure/azure-sdk-for-python

Community package: azure-ai-evaluation-openeval-adapter (evaluate() data/results <-> EvalPort interchange)

Open
#48,971 0 comments 1 reaction 0 assignees View on GitHub
Evaluation feature-request
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

Hi Azure AI Evaluation team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open, schema-validated JSON interchange format for portable LLM evaluation test cases, graders, suites, and results. It already has independently-tested adapter packages for ~20 other eval/observability frameworks (MLflow, LangSmith, Ragas, Vertex AI Gen AI Evaluation, Hugging Face `evaluate`, and others), and I built one for `azure-ai-evaluation` the same way.

This isn't a request for a change in this repo — I'm not proposing new API surface or asking for a design review, just flagging a working, tested community package in case it's useful to know about or link from docs.

**[`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter)**

```python
from azure.ai.evaluation import F1ScoreEvaluator, evaluate
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set

suite = to_openeval(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()}, suite_id="my_eval_suite")
assert validate_suite(suite).valid

result = evaluate(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()})
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```

`to_openeval()` accepts exactly what `evaluate()` itself accepts for `data`/`evaluators`, so it's a pure format bridge rather than new infrastructure. The one design choice worth flagging: every evaluator (local NLP metrics like F1/BLEU/ROUGE, AI-assisted evaluators needing a live `model_config`, and the content-safety evaluators needing a live Foundry project) maps to EvalPort's `custom` grader type rather than being force-fit into `semantic_similarity` or `llm_judge` — those types require params (`threshold`, `prompt`) this adapter can't honestly fabricate from the outside. Full mapping table and the flat-row parsing logic (recovering per-metric score/passed/reason from `evaluate()`'s real `outputs..*` column convention) are in the [README](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter#readme).

21 tests, all passing locally against the real installed `azure-ai-evaluation` package and EvalPort's real `validate_suite()`/`validate_result_set()` — not mocked.

No action needed — this lives entirely outside `azure-sdk-for-python` as an independent package (`pip install` via `git+`, not yet on PyPI). Flagging mainly for discoverability; happy to adjust the mapping if the evaluation module's public API shifts, or to send a one-line docs PR if there's a community-packages list this belongs on.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Contributor guide

Open the contributing guide

Research direction

Start with the linked azure-ai-evaluation-openeval-adapter README and check whether this repository has a community-packages list or other documentation entry point. No repository change is requested; if an appropriate listing exists, done would be a concise discoverability link, otherwise the issue needs clarification.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.