google / google/adk-python

to_openeval()/from_openeval(): interop between EvalSet and the EvalPort portable format

Abierto
#6,871 6 comentarios 0 reacciones 1 asignado Reclamado por @sanketpatil06 Ver en GitHub
eval needs review
Lenguaje dominante
Python
Estrellas
21.5k
Forks
4k
Merge medio
1 d 14 h
PR fusionados (30 d)
37

Descripción

Hi maintainers,

ADK's own description is "building, evaluating, and deploying" agents, and `google.adk.evaluation` is already one of the most complete eval subsystems I've looked at across agent SDKs — so this seemed like a particularly good fit to raise.

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one framework. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

Related precedent: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) is adding native `to_openeval()` / `from_openeval()` helpers onto that SDK's dataset/response types for the same reason.

Looking at `src/google/adk/evaluation/eval_case.py` and `eval_set.py`, an `EvalSet.eval_cases` is a `list[EvalCase]`, each with a `conversation: list[Invocation]` where `Invocation.user_content` / `final_response` are `genai_types.Content` and `intermediate_data` carries tool calls. You already ship `get_all_tool_calls()` for pulling tool-call trajectories out of that, which makes a converter pretty direct:

```python
# sketch, not a PR
def eval_case_to_openeval_testcase(case: EvalCase) -> dict:
inv = case.conversation[0] # simplest case: single-invocation, static conversation
return {
"id": case.eval_id,
"input": inv.user_content.parts[0].text if inv.user_content.parts else "",
"expected_output": (
inv.final_response.parts[0].text
if inv.final_response and inv.final_response.parts else None
),
"expected_tools": [tc.name for tc in get_all_tool_calls(inv.intermediate_data)],
"graders": ["exact_match"], # or derived from case.rubrics
}
```

which maps onto an EvalPort test case: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json

Since ADK already reads/writes eval sets as JSON via `local_eval_sets_manager.py`, this would mostly be a format bridge rather than new infrastructure — useful for anyone moving eval sets between ADK and other tooling. No pressure if this isn't a priority; happy to sketch a fuller PR if there's interest. (Per CONTRIBUTING.md, opening this as an issue first per your "large or complex changes" guidance.)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.