Azure-Samples / Azure-Samples/azureai-samples

Sample proposal: EvalPort ↔ azure-ai-evaluation interchange under scenarios/evaluate

Open Beginner friendly
#295 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
548
Forks
318
Avg merge
3d 15h
Merged PRs (30d)
1

Description

Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0), an open JSON spec for portable LLM evaluation datasets: test suites, test cases, graders, and result sets that can move between eval frameworks without losing meaning.

I looked through `scenarios/evaluate/` before opening this — the `Supported_Evaluation_Targets/` and `Supported_Evaluation_Metrics/` structure (and the comparison table in `scenarios/evaluate/README.md`) is a genuinely useful map of what `azure-ai-evaluation` supports, which is why I think this is worth a shot.

There's already a working, tested adapter for `azure-ai-evaluation` in EvalPort's `adapters/` directory: [`azure-ai-evaluation-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/azure-ai-evaluation-openeval-adapter). It's a small package with `to_openeval()` / `from_openeval()` / `evaluation_result_to_openeval()`, tested against the real `azure-ai-evaluation` package (21 tests, not mocks) — not a proposal, an existing reference implementation.

A minimal version of what a sample notebook would show:

```python
from azure.ai.evaluation import F1ScoreEvaluator
from azure_ai_evaluation_openeval_adapter import to_openeval, evaluation_result_to_openeval
from openeval.validate import validate_suite, validate_result_set

# Your evaluate()-shaped data + evaluators, exported as a portable EvalPort suite
suite = to_openeval(
data="my_eval_data.jsonl",
evaluators={"f1": F1ScoreEvaluator()},
suite_id="my_eval_suite",
)
assert validate_suite(suite).valid

# ...run azure.ai.evaluation.evaluate() as usual, then export the result:
result_set = evaluation_result_to_openeval(result, suite_id="my_eval_suite")
assert validate_result_set(result_set).valid
```

Would a small notebook like this be a welcome addition under `scenarios/evaluate/` (e.g. alongside `Supported_Evaluation_Targets/`)? It'd be scoped as one self-contained sample per your CONTRIBUTING.md guidance — happy to follow the README/notebook template if there's interest. No worries at all if this isn't a fit for the repo's current scope — just flagging since the adapter already exists and works.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Contributor guide

Open the contributing guide

Research direction

Start with scenarios/evaluate/README.md, the Supported_Evaluation_Targets/ and Supported_Evaluation_Metrics/ examples, and CONTRIBUTING.md for the notebook template. Use the existing azure-ai-evaluation adapter as the reference for the sample, and consider the work complete when a self-contained notebook demonstrates exporting a suite and result set and validates both.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, jupyter-notebook, python
Domain
ai, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.