a2aproject / a2aproject/A2A

[Feat]: Design for testability/eval

Open
#1,444 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
25.7k
Forks
2.6k
Avg merge
3d 6h
Merged PRs (30d)
16

Description

### Is your feature request related to a problem? Please describe.

I would like to discuss the current limitations regarding agent evaluation. While the ADK provides a simple evaluation mechanism, it cannot be easily extended to evaluate A2A agents.

Currently, the only viable method for this is to "patch" an agent's session with a modeled conversation prefix, issue a user message, and then analyze the model's response. However, patching the session in this manner is an unreliable workaround, as it depends on the non-guaranteed behavior of mapping a ContentID to a SessionID and the assumption that we have access to the session service.

I believe we should address this to provide developers with a more robust and extensible evaluation framework.

Best regards,
Eli Osherovich

### Describe the solution you'd like

A standard way to evaluate single- and multi-step conversations.

### Describe alternatives you've considered

Nothing specific at the moment. The obvious candidates are
* `SendMessageRequest` extended to allow inclusion of a conversation prefix.
* Session injection into `SendMessageRequest`

At the moment I consider only ADK, additional frameworks should be evaluated carefully to design a universal solution.

### Additional context

_No response_

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.