a2aproject / a2aproject/a2a-tck
[Feat]: Consider using scenarios in the SUT for complex tests
- Vorherrschende Sprache
- Python
- Sterne
- 50
- Forks
- 40
- Ø Merge
- 7 T. 1 Std.
- Gemergte PRs (30 T.)
- 1
Beschreibung
### Is your feature request related to a problem? Please describe.
At the moment, the TCK is actually used to test SDK (a2a-java, a2a-python), not Agents.
So these SDK provides "dummy agents" that are responds with "hello, word" messages but are not able to handle complex tasks.
This makes it very difficult to test complex scenarios possible with A2A. For example, we would want to test how a SUT handles the `TASK_STATE_INPUT_REQUIRED` but this requires the agent to reach that state.
### Describe the solution you'd like
There are different options to be able to perform these complex tests:
* Provide "guidelines" for SUT that they should comply to specific scenario ("when you receive message X, move to TASK_STATE_INPUT_REQUIRED, when you receive message Y, resume the task")
* Feed these guideline in an LLM and have the SUT wrap this LLM integration with their respective SDK.
As an analogy, this is how Jakarta EE specs are tested. The TCK has the code (Jakarta EE deployments) to run on app servers and the tests asserts their behaviours. Implementor of Jakarta EE specs are responsible for handling the deployments.
This would be similar:
* The TCK provides the scenarios
* The SUT are responsible for transcribing the scenarios (using their respective SDK APIs)
* The TCK calls the SUT and tests the scenarios.
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Beitragsleitfaden
Rechercherichtung
The issue discusses enhancing the TCK (Test Compatibility Kit) to test complex A2A agent scenarios, referencing SDKs like a2a-java and a2a-python. Start by examining the existing TCK structure and how dummy agents are currently implemented. Look for test files that handle task states like TASK_STATE_INPUT_REQUIRED to understand the current limitations. Research Jakarta EE TCK analogies for scenario-based testing patterns.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java, python
- Bereich
- backend-api-design, testing
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 30/100