Proposal: agent-evaluation-openeval-adapter for portable Test/TestSuite/TestResult interchange

Hi maintainers, thanks for building Agent Evaluation — the `steps`/`expected_results`-driven multi-turn test format is a clean way to spec agent behavior.

**What I'd like to propose:** a small, separate adapter package (`agent-evaluation-openeval-adapter`) that converts between Agent Evaluation's native models and [EvalPort](https://github.com/adhabnr-ux/evalport), an open JSON interchange format for eval datasets (a `Suite` of test cases + graders, and a `ResultSet` of per-test-case results + grader results — [spec](https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md)). The goal is portability: a `TestSuite` written for Agent Evaluation could be exported once and replayed against other eval/observability tooling (and vice versa), without the framework itself taking on that dependency.

This looks feasible because Agent Evaluation already models test config and results as real objects rather than bare pass/fail floats:

- `agenteval.test.Test` (`src/agenteval/test/test.py`) — pydantic `BaseModel` with `name`, `steps: list[str]`, `expected_results: list[str]`, `initial_prompt`, `max_turns`, `hook` — maps naturally onto an EvalPort test case.
- `agenteval.test.TestSuite` (`src/agenteval/test/test_suite.py`) — collection of `Test`s with `TestSuite.load(config, filter)` — maps onto an EvalPort `Suite`.
- `agenteval.conversation.Conversation` — `messages: list[(role, str)]` + `turns` — maps onto an EvalPort transcript.
- `agenteval.test.TestResult` (`src/agenteval/test/test_result.py`) — pydantic `BaseModel` with `test_name`, `result`, `reasoning`, `passed: bool`, `conversation: Conversation` — maps onto an EvalPort `ResultSet` entry + grader result.

Rough sketch of what the adapter would look like (kept out of core `agenteval`, in its own package/repo under the EvalPort org, following the pattern of the other 36 adapters already in that ecosystem):

```python
# agent_evaluation_openeval_adapter/convert.py
from agenteval.test import Test, TestSuite, TestResult
from agenteval.conversation import Conversation
from openeval.spec import Suite, TestCase, ResultSet, GraderResult

def suite_to_openeval(suite: TestSuite) -> Suite:
return Suite(
test_cases=[
TestCase(
id=t.name,
input=t.initial_prompt,
steps=t.steps,
expected=t.expected_results,
metadata={"max_turns": t.max_turns, "hook": t.hook},
)
for t in suite
]
)

def suite_from_openeval(suite: Suite) -> TestSuite:
return TestSuite(
tests=[
Test(
name=tc.id,
steps=tc.steps,
expected_results=tc.expected,
initial_prompt=tc.input,
max_turns=(tc.metadata or {}).get("max_turns", 2),
hook=(tc.metadata or {}).get("hook"),
)
for tc in suite.test_cases
]
)

def test_result_to_openeval(result: TestResult) -> GraderResult:
return GraderResult(
test_case_id=result.test_name,
passed=result.passed,
reasoning=result.reasoning,
transcript=[
{"role": role, "content": msg} for role, msg in result.conversation
],
)
```

Happy to build and maintain this as a standalone package rather than a PR into this repo, so it stays optional and doesn't add a dependency to core `agenteval`. Opening this issue first per the contributing guide ("open an issue to discuss any significant work") — mainly to (a) sanity-check that `Test`/`TestSuite`/`TestResult`/`Conversation` are the right integration points and stable enough to build against, and (b) ask whether you'd want it listed/linked from your docs once it exists, or whether you'd rather it stay purely external. No PR attached to this repo either way — this issue is just to flag the idea and get a read from maintainers before I sink time into it.

Reference: EvalPort spec — https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Start by reading src/agenteval/test/test.py, src/agenteval/test/test_suite.py, src/agenteval/test/test_result.py, and the Conversation model, then compare their fields with the linked EvalPort specification. Confirm with maintainers whether these are stable integration points and whether the standalone adapter should be linked from the documentation; done means an agreed scope and external package plan, rather than a change in this repository.

Written by the indexing model from the issue text.