a2aproject / a2aproject/a2a-tck
[Feat]: Consider using scenarios in the SUT for complex tests
- Dominant language
- Python
- Stars
- 50
- Forks
- 40
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 1
Description
### Is your feature request related to a problem? Please describe.
At the moment, the TCK is actually used to test SDK (a2a-java, a2a-python), not Agents.
So these SDK provides "dummy agents" that are responds with "hello, word" messages but are not able to handle complex tasks.
This makes it very difficult to test complex scenarios possible with A2A. For example, we would want to test how a SUT handles the `TASK_STATE_INPUT_REQUIRED` but this requires the agent to reach that state.
### Describe the solution you'd like
There are different options to be able to perform these complex tests:
* Provide "guidelines" for SUT that they should comply to specific scenario ("when you receive message X, move to TASK_STATE_INPUT_REQUIRED, when you receive message Y, resume the task")
* Feed these guideline in an LLM and have the SUT wrap this LLM integration with their respective SDK.
As an analogy, this is how Jakarta EE specs are tested. The TCK has the code (Jakarta EE deployments) to run on app servers and the tests asserts their behaviours. Implementor of Jakarta EE specs are responsible for handling the deployments.
This would be similar:
* The TCK provides the scenarios
* The SUT are responsible for transcribing the scenarios (using their respective SDK APIs)
* The TCK calls the SUT and tests the scenarios.
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.