[Feat]: Design for testability/eval
- Dominant language
- Shell
- Stars
- 25.7k
- Forks
- 2.6k
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 16
Description
### Is your feature request related to a problem? Please describe.
I would like to discuss the current limitations regarding agent evaluation. While the ADK provides a simple evaluation mechanism, it cannot be easily extended to evaluate A2A agents.
Currently, the only viable method for this is to "patch" an agent's session with a modeled conversation prefix, issue a user message, and then analyze the model's response. However, patching the session in this manner is an unreliable workaround, as it depends on the non-guaranteed behavior of mapping a ContentID to a SessionID and the assumption that we have access to the session service.
I believe we should address this to provide developers with a more robust and extensible evaluation framework.
Best regards,
Eli Osherovich
### Describe the solution you'd like
A standard way to evaluate single- and multi-step conversations.
### Describe alternatives you've considered
Nothing specific at the moment. The obvious candidates are
* `SendMessageRequest` extended to allow inclusion of a conversation prefix.
* Session injection into `SendMessageRequest`
At the moment I consider only ADK, additional frameworks should be evaluated carefully to design a universal solution.
### Additional context
_No response_
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.