Tool Evaluations: Add Vally evaluation harness for Event Hubs tools
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Implemented functionality
PR #3159 adds the initial [Vally](https://microsoft.github.io/vally) evaluation harness for Azure Event Hubs under `servers/Azure.Mcp.Server/tests/Vally`.
The harness currently evaluates these Event Hubs operations:
- Event Hubs namespaces: get, update, and delete
- Event hubs: get, update, and delete
- Consumer groups: get, update, and delete
Each evaluation is defined by a shared `.eval.yaml` grading specification and a corresponding `.experiment.yaml` prompt set. The runner executes baseline and treatment variants so the server's contribution can be isolated, using outcome-based LLM-judge graders.
`Invoke-VallyEval.ps1` discovers evaluations by area and tool. Event Hubs resource scripts provision and remove the required test resources, including the Safe Secrets Standard configuration and `DeleteAfter` tagging. The harness also includes documentation and the Event Hubs Bicep templates needed to support these evaluations.
Implemented in [PR #3159](https://github.com/microsoft/mcp/pull/3159).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.