google / google/adk-python

to_openeval()/from_openeval(): interop between EvalSet and the EvalPort portable format

Open
#6,871 6 comments 0 reactions 1 assignee Claimed by @sanketpatil06 View on GitHub
eval needs review
Dominant language
Python
Stars
21.5k
Forks
4k
Avg merge
1d 14h
Merged PRs (30d)
37

Description

Hi maintainers,

ADK's own description is "building, evaluating, and deploying" agents, and `google.adk.evaluation` is already one of the most complete eval subsystems I've looked at across agent SDKs — so this seemed like a particularly good fit to raise.

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one framework. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

Related precedent: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) is adding native `to_openeval()` / `from_openeval()` helpers onto that SDK's dataset/response types for the same reason.

Looking at `src/google/adk/evaluation/eval_case.py` and `eval_set.py`, an `EvalSet.eval_cases` is a `list[EvalCase]`, each with a `conversation: list[Invocation]` where `Invocation.user_content` / `final_response` are `genai_types.Content` and `intermediate_data` carries tool calls. You already ship `get_all_tool_calls()` for pulling tool-call trajectories out of that, which makes a converter pretty direct:

```python
# sketch, not a PR
def eval_case_to_openeval_testcase(case: EvalCase) -> dict:
inv = case.conversation[0] # simplest case: single-invocation, static conversation
return {
"id": case.eval_id,
"input": inv.user_content.parts[0].text if inv.user_content.parts else "",
"expected_output": (
inv.final_response.parts[0].text
if inv.final_response and inv.final_response.parts else None
),
"expected_tools": [tc.name for tc in get_all_tool_calls(inv.intermediate_data)],
"graders": ["exact_match"], # or derived from case.rubrics
}
```

which maps onto an EvalPort test case: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json

Since ADK already reads/writes eval sets as JSON via `local_eval_sets_manager.py`, this would mostly be a format bridge rather than new infrastructure — useful for anyone moving eval sets between ADK and other tooling. No pressure if this isn't a priority; happy to sketch a fuller PR if there's interest. (Per CONTRIBUTING.md, opening this as an issue first per your "large or complex changes" guidance.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.