Azure-Samples / Azure-Samples/azureai-samples
Sample proposal: EvalPort ↔ azure-ai-evaluation interchange under scenarios/evaluate
- Dominant language
- Jupyter Notebook
- Stars
- 548
- Forks
- 318
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 1
Description
Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open JSON spec for portable LLM evaluation datasets: test suites, test cases, graders, and result sets that can move between eval frameworks without losing meaning.
I looked through `scenarios/evaluate/` before opening this — the `Supported_Evaluation_Targets/` and `Supported_Evaluation_Metrics/` structure (and the comparison table in `scenarios/evaluate/README.md`) is a genuinely useful map of what `azure-ai-evaluation` supports, which is why I think this is worth a shot.
There's already a working, tested adapter for `azure-ai-evaluation` in EvalPort's `adapters/` directory: [`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter). It's a small package with `to_openeval()` / `from_openeval()` / `evaluation_result_to_openeval()`, tested against the real `azure-ai-evaluation` package (21 tests, not mocks) — not a proposal, an existing reference implementation.
A minimal version of what a sample notebook would show:
```python
from azure.ai.evaluation import F1ScoreEvaluator
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set
# Your evaluate()-shaped data + evaluators, exported as a portable EvalPort suite
suite = to_openeval(
data="my_eval_data.jsonl",
evaluators={"f1": F1ScoreEvaluator()},
suite_id="my_eval_suite",
)
assert validate_suite(suite).valid
# ...run azure.ai.evaluation.evaluate() as usual, then export the result:
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```
Would a small notebook like this be a welcome addition under `scenarios/evaluate/` (e.g. alongside `Supported_Evaluation_Targets/`)? It'd be scoped as one self-contained sample per your CONTRIBUTING.md guidance — happy to follow the README/notebook template if there's interest. No worries at all if this isn't a fit for the repo's current scope — just flagging since the adapter already exists and works.
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Contributor guide
Research direction
Start with scenarios/evaluate/README.md, the Supported_Evaluation_Targets/ and Supported_Evaluation_Metrics/ examples, and CONTRIBUTING.md for the notebook template. Use the existing azure-ai-evaluation adapter as the reference for the sample, and consider the work complete when a self-contained notebook demonstrates exporting a suite and result set and validates both.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, jupyter-notebook, python
- Domain
- ai, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100