In-Conversation Evals
- Lenguaje dominante
- Python
- Estrellas
- 21.5k
- Forks
- 4k
- Merge medio
- 1 d 14 h
- PR fusionados (30 d)
- 37
Descripción
**Is your feature request related to a problem? Please describe.**
The current Vertex AI Evaluation API is primarily designed to evaluate the full end-to-end trajectory of an agent's response. This poses a significant challenge when attempting to get fine-grained feedback on individual steps within a complex agent's chain of thought. Evaluating only the final output can lead to high costs and obscure the specific steps that may be failing or succeeding, making it difficult to identify opportunities for improvement. The inability to check intermediate responses prevents a "fail-fast" approach and hinders the efficient creation of high-quality examples for an ExampleStore.
**Describe the solution you'd like**
I would like to propose adding an optional `evaluation_parameter` to the `Agent` or `LlmAgent` classes within Agent Development Kit (ADK). This parameter would accept a reference to a function or an object that contains a specific evaluation logic. This evaluation would be executed immediately after the agent generates an individual response (whether with or without a tool call), but before the full trajectory is completed.
This would allow for the following:
* **Granular Feedback:** Developers could get immediate feedback on a per-step basis, identifying problematic responses in a multi-step process.
* **Cost Savings:** By evaluating and potentially halting a trajectory early, costs associated with running a full, potentially flawed, end-to-end evaluation can be saved.
* **Targeted ExampleStore Population:** The per-step evaluation provides the signal needed to efficiently build an ExampleStore, as a high-quality intermediate response could be flagged for inclusion, directly incorporating exemplary work into the agent system.
**Alternatives considered**
One alternative is to use a `CustomAgent` that encapsulates the specific evaluation logic. While this is a technically viable workaround, it adds architectural complexity. It requires creating a new agent class for each distinct evaluation need, rather than leveraging a native, in-process evaluation capability within the ADK itself. The proposed solution would be a much cleaner and more integrated approach.
**Additional context**
The ability to affirm if a model is performant with its task following and response before evaluating the entire end-to-end process is a crucial capability for professional-grade agent development. This feature would be a significant addition to the ADK, enabling more robust, cost-effective, and data-driven agent development cycles.
*Links*
* [ADK | Evaluate](https://google.github.io/adk-docs/evaluate/)
* [Vertex AI | Evaluate GenAI Agents](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-agents) - currently the same functionality as the link above
* [Vertex AI | Evaluate an agent with Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/evaluate#custom)
* [Vertex AI | Evaluation API](https://cloud.google.com/vertex-ai/generative-ai/docs/models/run-evaluation)
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.