to_openeval()/from_openeval(): interop between EvalSet and the EvalPort portable format
- Vorherrschende Sprache
- Python
- Sterne
- 21.5k
- Forks
- 4k
- Ø Merge
- 1 T. 14 Std.
- Gemergte PRs (30 T.)
- 37
Beschreibung
Hi maintainers,
ADK's own description is "building, evaluating, and deploying" agents, and `google.adk.evaluation` is already one of the most complete eval subsystems I've looked at across agent SDKs — so this seemed like a particularly good fit to raise.
I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one framework. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
Related precedent: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) is adding native `to_openeval()` / `from_openeval()` helpers onto that SDK's dataset/response types for the same reason.
Looking at `src/google/adk/evaluation/eval_case.py` and `eval_set.py`, an `EvalSet.eval_cases` is a `list[EvalCase]`, each with a `conversation: list[Invocation]` where `Invocation.user_content` / `final_response` are `genai_types.Content` and `intermediate_data` carries tool calls. You already ship `get_all_tool_calls()` for pulling tool-call trajectories out of that, which makes a converter pretty direct:
```python
# sketch, not a PR
def eval_case_to_openeval_testcase(case: EvalCase) -> dict:
inv = case.conversation[0] # simplest case: single-invocation, static conversation
return {
"id": case.eval_id,
"input": inv.user_content.parts[0].text if inv.user_content.parts else "",
"expected_output": (
inv.final_response.parts[0].text
if inv.final_response and inv.final_response.parts else None
),
"expected_tools": [tc.name for tc in get_all_tool_calls(inv.intermediate_data)],
"graders": ["exact_match"], # or derived from case.rubrics
}
```
which maps onto an EvalPort test case: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json
Since ADK already reads/writes eval sets as JSON via `local_eval_sets_manager.py`, this would mostly be a format bridge rather than new infrastructure — useful for anyone moving eval sets between ADK and other tooling. No pressure if this isn't a priority; happy to sketch a fuller PR if there's interest. (Per CONTRIBUTING.md, opening this as an issue first per your "large or complex changes" guidance.)
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.