getsentry / getsentry/sentry-mcp

Migrate evals off the legacy vitest-evals scorer API

Open
#967 2 comments 0 reactions 0 assignees View on GitHub
dependencies javascript
Dominant language
TypeScript
Stars
853
Forks
144
Avg merge
19h 25m
Merged PRs (30d)
32

Description

The eval suite is still using `vitest-evals@0.4.0` and the legacy scorer-first API. The request is to migrate `packages/mcp-server-evals` to the current harness-first `vitest-evals` API.

## Current State

- `pnpm-workspace.yaml` catalogs `vitest-evals: ^0.4.0`, and the lockfile resolves it to `0.4.0`.
- `packages/mcp-server-evals` depends on `vitest-evals` through the catalog.
- Eval files under `packages/mcp-server-evals/src/evals` import `describeEval` from `vitest-evals` and use `scorers: [...]`.
- Several agent evals use `ToolCallScorer` from `vitest-evals`.
- `packages/mcp-server-evals/src/evals/utils/toolPredictionScorer.ts` defines a custom `ToolPredictionScorer`.

## Migration Scope

- Bump `vitest-evals` from `0.4.0` to the current target version.
- npm currently reports `0.8.0` as `latest` and `0.9.0-beta.3` as `next`.
- Migrate eval suites from scorer-first `describeEval` usage to the harness-first API.
- Replace legacy scorer usage with judge-based equivalents where practical.
- Replace or remove the custom `ToolPredictionScorer` if the newer package provides suitable built-in judge behavior.

## Notes

- Reporter wiring appears to already use `vitest-evals/reporter` in both `packages/mcp-server-evals/package.json` and `packages/mcp-server-evals/vitest.config.ts`.
- A targeted package build was attempted, but it failed on unresolved workspace package imports/types in the isolated build path, not on the `vitest-evals` migration itself.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.