microsoft / microsoft/agent-framework

Python: [Feature]: EvalPort import/export support (to_openeval() / from_openeval())

Open
#7,590 0 comments 0 reactions 1 assignee View on GitHub

@sphenry is already working on this.

Since Aug 10, 2026.

python
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

### Description

## Overview

We maintain [EvalPort](https://github.com/adhabnr-ux/evalport) (v1.0.0), an open standard (Apache 2.0) for portable LLM evaluation datasets — test cases, graders, eval suites, and result sets in a single JSON format, with converters and SDKs (TypeScript + Python) so evals aren't locked to one framework or vendor.

We originally proposed this for AutoGen (microsoft/autogen#8005), and a contributor built a draft adapter there (microsoft/autogen#8009). Since AutoGen has since moved to maintenance mode (bug fixes / security / docs only, no new features), we published that work as a standalone package instead: https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter. Since Agent Framework is AutoGen's actively developed successor, we'd like to propose native EvalPort support here instead of another bolt-on package.

## What We're Proposing

Add `to_openeval()` / `from_openeval()` conversion functions (roughly 50-100 lines using our SDK) so Agent Framework eval results can be exported to EvalPort's JSON format, and EvalPort suites can be imported as Agent Framework eval tasks. That gets round-tripping with DeepEval, Promptfoo, Inspect AI, and any other EvalPort-compatible tool for free.

Happy to align on the exact module location / package boundaries with the team, same as we did on the AutoGen PR.

## Resources

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Python SDK: pip install evalport-sdk (imports as openeval for backward compatibility)
npm SDK: npm install evalport-sdk
Reference adapter with the same shape (AutoGen): https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter

We'd love to collaborate — happy to open a draft PR ourselves, or review one if someone on the team wants to take it on.

### Code Sample

```markdown
from openeval.types import OPENEVAL_VERSION

def to_openeval(agent_framework_eval_result) -> dict:
"""Export Agent Framework eval results to an EvalPort suite."""
test_cases = []
for result in agent_framework_eval_result.results:
test_cases.append({
"id": result.task_id,
"input": result.task_description,
"expected_output": result.expected_output,
"graders": ["gr_output_match"],
})
return {
"version": OPENEVAL_VERSION,
"id": f"agent_framework_eval_{agent_framework_eval_result.run_id}",
"test_cases": test_cases,
"graders": [{"id": "gr_output_match", "type": "exact_match", "params": {"ignore_case": True}}],
}

def from_openeval(suite: dict) -> list:
"""Import an EvalPort suite into Agent Framework eval tasks."""
return [
{"task_id": tc["id"], "description": tc["input"], "expected_output": tc.get("expected_output", "")}
for tc in suite.get("test_cases", [])
]
```

### Language/SDK

Both

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.