microsoft / microsoft/agent-framework
Python: [Feature]: EvalPort import/export support (to_openeval() / from_openeval())
@sphenry is already working on this.
Since Aug 10, 2026.
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
### Description
## Overview
We maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (v1.0.0), an open standard (Apache 2.0) for portable LLM evaluation datasets — test cases, graders, eval suites, and result sets in a single JSON format, with converters and SDKs (TypeScript + Python) so evals aren't locked to one framework or vendor.
We originally proposed this for AutoGen (microsoft/autogen#8005), and a contributor built a draft adapter there (microsoft/autogen#8009). Since AutoGen has since moved to maintenance mode (bug fixes / security / docs only, no new features), we published that work as a standalone package instead: https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter. Since Agent Framework is AutoGen's actively developed successor, we'd like to propose native EvalPort support here instead of another bolt-on package.
## What We're Proposing
Add `to_openeval()` / `from_openeval()` conversion functions (roughly 50-100 lines using our SDK) so Agent Framework eval results can be exported to EvalPort's JSON format, and EvalPort suites can be imported as Agent Framework eval tasks. That gets round-tripping with DeepEval, Promptfoo, Inspect AI, and any other EvalPort-compatible tool for free.
Happy to align on the exact module location / package boundaries with the team, same as we did on the AutoGen PR.
## Resources
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Python SDK: pip install evalport-sdk (imports as openeval for backward compatibility)
npm SDK: npm install evalport-sdk
Reference adapter with the same shape (AutoGen): https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter
We'd love to collaborate — happy to open a draft PR ourselves, or review one if someone on the team wants to take it on.
### Code Sample
```markdown
from openeval.types import OPENEVAL_VERSION
def to_openeval(agent_framework_eval_result) -> dict:
"""Export Agent Framework eval results to an EvalPort suite."""
test_cases = []
for result in agent_framework_eval_result.results:
test_cases.append({
"id": result.task_id,
"input": result.task_description,
"expected_output": result.expected_output,
"graders": ["gr_output_match"],
})
return {
"version": OPENEVAL_VERSION,
"id": f"agent_framework_eval_{agent_framework_eval_result.run_id}",
"test_cases": test_cases,
"graders": [{"id": "gr_output_match", "type": "exact_match", "params": {"ignore_case": True}}],
}
def from_openeval(suite: dict) -> list:
"""Import an EvalPort suite into Agent Framework eval tasks."""
return [
{"task_id": tc["id"], "description": tc["input"], "expected_output": tc.get("expected_output", "")}
for tc in suite.get("test_cases", [])
]
```
### Language/SDK
Both
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.