Rewrite Vally evaluator using the TypeScript SDK
@conniey is already working on this.
Since Jul 31, 2026.
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Description
Rewrite the Vally evaluation tooling introduced in #3163 in TypeScript and use the Vally SDK's native schema types instead of duplicating the stimulus and grader configuration models in C#.
The current C# implementation clones Vally schema structures, which creates maintenance risk and can drift as the SDK schema evolves. Using the TypeScript SDK directly should keep generated evaluation specifications aligned with Vally.
## Acceptance criteria
- Replace the C# Vally evaluator and duplicated schema models with a TypeScript implementation that uses the Vally SDK.
- Preserve generation of evaluation specifications from existing end-to-end prompts and build metadata.
- Preserve the current workflow and local invocation behavior introduced in #3163.
- Add or update automated tests for the TypeScript implementation.
- Update the contributor documentation and build integration as needed.
## Context
Follow-up from https://github.com/microsoft/mcp/pull/3163#discussion_r3669526750.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.