google / google/adk-python

In-Conversation Evals

未关闭
#2,746 1 条评论 0 个 reaction 已指派 1 人 已被 @ankursharmas 认领 在 GitHub 查看
eval needs review
主要语言
Python
星标
21.5k
派生
4k
平均合并
1 天 14 小时
30 天内合并 PR
37

描述

**Is your feature request related to a problem? Please describe.**
The current Vertex AI Evaluation API is primarily designed to evaluate the full end-to-end trajectory of an agent's response. This poses a significant challenge when attempting to get fine-grained feedback on individual steps within a complex agent's chain of thought. Evaluating only the final output can lead to high costs and obscure the specific steps that may be failing or succeeding, making it difficult to identify opportunities for improvement. The inability to check intermediate responses prevents a "fail-fast" approach and hinders the efficient creation of high-quality examples for an ExampleStore.

**Describe the solution you'd like**
I would like to propose adding an optional `evaluation_parameter` to the `Agent` or `LlmAgent` classes within Agent Development Kit (ADK). This parameter would accept a reference to a function or an object that contains a specific evaluation logic. This evaluation would be executed immediately after the agent generates an individual response (whether with or without a tool call), but before the full trajectory is completed.

This would allow for the following:

* **Granular Feedback:** Developers could get immediate feedback on a per-step basis, identifying problematic responses in a multi-step process.
* **Cost Savings:** By evaluating and potentially halting a trajectory early, costs associated with running a full, potentially flawed, end-to-end evaluation can be saved.
* **Targeted ExampleStore Population:** The per-step evaluation provides the signal needed to efficiently build an ExampleStore, as a high-quality intermediate response could be flagged for inclusion, directly incorporating exemplary work into the agent system.

**Alternatives considered**
One alternative is to use a `CustomAgent` that encapsulates the specific evaluation logic. While this is a technically viable workaround, it adds architectural complexity. It requires creating a new agent class for each distinct evaluation need, rather than leveraging a native, in-process evaluation capability within the ADK itself. The proposed solution would be a much cleaner and more integrated approach.

**Additional context**
The ability to affirm if a model is performant with its task following and response before evaluating the entire end-to-end process is a crucial capability for professional-grade agent development. This feature would be a significant addition to the ADK, enabling more robust, cost-effective, and data-driven agent development cycles.

*Links*
* [ADK | Evaluate](https://google.github.io/adk-docs/evaluate/)
* [Vertex AI | Evaluate GenAI Agents](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-agents) - currently the same functionality as the link above
* [Vertex AI | Evaluate an agent with Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/evaluate#custom)
* [Vertex AI | Evaluation API](https://cloud.google.com/vertex-ai/generative-ai/docs/models/run-evaluation)

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。