microsoft / microsoft/Power-CAT-Copilot-Studio-Kit

Agent Test Run Validation Only Runs On 1 Output Message

Open
#517 0 comments 0 reactions 1 assignee View on GitHub

@psimolin is already working on this.

Since Dec 12, 2025.

enhancement
Dominant language
TypeScript
Stars
450
Forks
95
Avg merge
26m
Merged PRs (30d)
5

Description

Today, if an agent provides multiple messages in the output for an input query, the framework either takes the first message or the message at position X ( X being defined in the test beforehand ). This message is stored in cat_response attribute.

But the framework does get the full response and stores it in cat_actualcompleteresponse attribute.

It is not always possible to know the position of the exact message we want to validate beforehand, it can change dynamically according to the configurations of the agent.

For a simple Response Match type test, the framework relies on the cat_response attribute to validate.
Ask is that it should perform the checks on cat_actualcompleteresponse attribute if there is no message position X defined in the test. It should look at the complete response.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.