to_openeval()/from_openeval(): interop between EvalSet and the EvalPort portable format
- Lenguaje dominante
- Python
- Estrellas
- 21.5k
- Forks
- 4k
- Merge medio
- 1 d 14 h
- PR fusionados (30 d)
- 37
Descripción
Hi maintainers,
ADK's own description is "building, evaluating, and deploying" agents, and `google.adk.evaluation` is already one of the most complete eval subsystems I've looked at across agent SDKs — so this seemed like a particularly good fit to raise.
I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one framework. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
Related precedent: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) is adding native `to_openeval()` / `from_openeval()` helpers onto that SDK's dataset/response types for the same reason.
Looking at `src/google/adk/evaluation/eval_case.py` and `eval_set.py`, an `EvalSet.eval_cases` is a `list[EvalCase]`, each with a `conversation: list[Invocation]` where `Invocation.user_content` / `final_response` are `genai_types.Content` and `intermediate_data` carries tool calls. You already ship `get_all_tool_calls()` for pulling tool-call trajectories out of that, which makes a converter pretty direct:
```python
# sketch, not a PR
def eval_case_to_openeval_testcase(case: EvalCase) -> dict:
inv = case.conversation[0] # simplest case: single-invocation, static conversation
return {
"id": case.eval_id,
"input": inv.user_content.parts[0].text if inv.user_content.parts else "",
"expected_output": (
inv.final_response.parts[0].text
if inv.final_response and inv.final_response.parts else None
),
"expected_tools": [tc.name for tc in get_all_tool_calls(inv.intermediate_data)],
"graders": ["exact_match"], # or derived from case.rubrics
}
```
which maps onto an EvalPort test case: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json
Since ADK already reads/writes eval sets as JSON via `local_eval_sets_manager.py`, this would mostly be a format bridge rather than new infrastructure — useful for anyone moving eval sets between ADK and other tooling. No pressure if this isn't a priority; happy to sketch a fuller PR if there's interest. (Per CONTRIBUTING.md, opening this as an issue first per your "large or complex changes" guidance.)
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.