google / google/adk-python

to_openeval()/from_openeval(): interop between EvalSet and the EvalPort portable format

未关闭
#6,871 6 条评论 0 个 reaction 已指派 1 人 已被 @sanketpatil06 认领 在 GitHub 查看
eval needs review
主要语言
Python
星标
21.5k
派生
4k
平均合并
1 天 14 小时
30 天内合并 PR
37

描述

Hi maintainers,

ADK's own description is "building, evaluating, and deploying" agents, and `google.adk.evaluation` is already one of the most complete eval subsystems I've looked at across agent SDKs — so this seemed like a particularly good fit to raise.

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one framework. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

Related precedent: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) is adding native `to_openeval()` / `from_openeval()` helpers onto that SDK's dataset/response types for the same reason.

Looking at `src/google/adk/evaluation/eval_case.py` and `eval_set.py`, an `EvalSet.eval_cases` is a `list[EvalCase]`, each with a `conversation: list[Invocation]` where `Invocation.user_content` / `final_response` are `genai_types.Content` and `intermediate_data` carries tool calls. You already ship `get_all_tool_calls()` for pulling tool-call trajectories out of that, which makes a converter pretty direct:

```python
# sketch, not a PR
def eval_case_to_openeval_testcase(case: EvalCase) -> dict:
inv = case.conversation[0] # simplest case: single-invocation, static conversation
return {
"id": case.eval_id,
"input": inv.user_content.parts[0].text if inv.user_content.parts else "",
"expected_output": (
inv.final_response.parts[0].text
if inv.final_response and inv.final_response.parts else None
),
"expected_tools": [tc.name for tc in get_all_tool_calls(inv.intermediate_data)],
"graders": ["exact_match"], # or derived from case.rubrics
}
```

which maps onto an EvalPort test case: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json

Since ADK already reads/writes eval sets as JSON via `local_eval_sets_manager.py`, this would mostly be a format bridge rather than new infrastructure — useful for anyone moving eval sets between ADK and other tooling. No pressure if this isn't a priority; happy to sketch a fuller PR if there's interest. (Per CONTRIBUTING.md, opening this as an issue first per your "large or complex changes" guidance.)

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。