microsoft / microsoft/Power-CAT-Copilot-Studio-Kit
Agent Test Run Validation Only Runs On 1 Output Message
@psimolin is already working on this.
Since Dec 12, 2025.
- Dominant language
- TypeScript
- Stars
- 450
- Forks
- 95
- Avg merge
- 26m
- Merged PRs (30d)
- 5
Description
Today, if an agent provides multiple messages in the output for an input query, the framework either takes the first message or the message at position X ( X being defined in the test beforehand ). This message is stored in cat_response attribute.
But the framework does get the full response and stores it in cat_actualcompleteresponse attribute.
It is not always possible to know the position of the exact message we want to validate beforehand, it can change dynamically according to the configurations of the agent.
For a simple Response Match type test, the framework relies on the cat_response attribute to validate.
Ask is that it should perform the checks on cat_actualcompleteresponse attribute if there is no message position X defined in the test. It should look at the complete response.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.