Azure / Azure/azure-sdk-for-python

Community package: azure-ai-evaluation-openeval-adapter (evaluate() data/results <-> EvalPort interchange)

Abierto
#48,971 0 comentarios 1 reacción 0 asignados Ver en GitHub
Evaluation feature-request
Lenguaje dominante
Python
Estrellas
5.6k
Forks
3.4k
Merge medio
2 d 2 h
PR fusionados (30 d)
213

Descripción

Hi Azure AI Evaluation team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open, schema-validated JSON interchange format for portable LLM evaluation test cases, graders, suites, and results. It already has independently-tested adapter packages for ~20 other eval/observability frameworks (MLflow, LangSmith, Ragas, Vertex AI Gen AI Evaluation, Hugging Face `evaluate`, and others), and I built one for `azure-ai-evaluation` the same way.

This isn't a request for a change in this repo — I'm not proposing new API surface or asking for a design review, just flagging a working, tested community package in case it's useful to know about or link from docs.

**[`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter)**

```python
from azure.ai.evaluation import F1ScoreEvaluator, evaluate
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set

suite = to_openeval(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()}, suite_id="my_eval_suite")
assert validate_suite(suite).valid

result = evaluate(data="my_eval_data.jsonl", evaluators={"f1": F1ScoreEvaluator()})
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```

`to_openeval()` accepts exactly what `evaluate()` itself accepts for `data`/`evaluators`, so it's a pure format bridge rather than new infrastructure. The one design choice worth flagging: every evaluator (local NLP metrics like F1/BLEU/ROUGE, AI-assisted evaluators needing a live `model_config`, and the content-safety evaluators needing a live Foundry project) maps to EvalPort's `custom` grader type rather than being force-fit into `semantic_similarity` or `llm_judge` — those types require params (`threshold`, `prompt`) this adapter can't honestly fabricate from the outside. Full mapping table and the flat-row parsing logic (recovering per-metric score/passed/reason from `evaluate()`'s real `outputs..*` column convention) are in the [README](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter#readme).

21 tests, all passing locally against the real installed `azure-ai-evaluation` package and EvalPort's real `validate_suite()`/`validate_result_set()` — not mocked.

No action needed — this lives entirely outside `azure-sdk-for-python` as an independent package (`pip install` via `git+`, not yet on PyPI). Flagging mainly for discoverability; happy to adjust the mapping if the evaluation module's public API shifts, or to send a one-line docs PR if there's a community-packages list this belongs on.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Empieza por el README enlazado de azure-ai-evaluation-openeval-adapter y comprueba si este repositorio tiene una lista de community-packages u otro punto de entrada a la documentación. No se solicita ningún cambio en el repositorio; si existe una lista adecuada, done sería un enlace conciso para mejorar su descubribilidad; de lo contrario, es necesario aclarar el issue.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
azure, python
Área
documentation
Tipo de issue
Documentación
Dificultad
1/5
Tiempo estimado
Menos de una hora
Estado de actividad
Activo
Claridad
Necesita aclaración
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.