getsentry / getsentry/sentry-mcp
Migrate evals off the legacy vitest-evals scorer API
- Dominant language
- TypeScript
- Stars
- 853
- Forks
- 144
- Avg merge
- 19h 25m
- Merged PRs (30d)
- 32
Description
The eval suite is still using `vitest-evals@0.4.0` and the legacy scorer-first API. The request is to migrate `packages/mcp-server-evals` to the current harness-first `vitest-evals` API.
## Current State
- `pnpm-workspace.yaml` catalogs `vitest-evals: ^0.4.0`, and the lockfile resolves it to `0.4.0`.
- `packages/mcp-server-evals` depends on `vitest-evals` through the catalog.
- Eval files under `packages/mcp-server-evals/src/evals` import `describeEval` from `vitest-evals` and use `scorers: [...]`.
- Several agent evals use `ToolCallScorer` from `vitest-evals`.
- `packages/mcp-server-evals/src/evals/utils/toolPredictionScorer.ts` defines a custom `ToolPredictionScorer`.
## Migration Scope
- Bump `vitest-evals` from `0.4.0` to the current target version.
- npm currently reports `0.8.0` as `latest` and `0.9.0-beta.3` as `next`.
- Migrate eval suites from scorer-first `describeEval` usage to the harness-first API.
- Replace legacy scorer usage with judge-based equivalents where practical.
- Replace or remove the custom `ToolPredictionScorer` if the newer package provides suitable built-in judge behavior.
## Notes
- Reporter wiring appears to already use `vitest-evals/reporter` in both `packages/mcp-server-evals/package.json` and `packages/mcp-server-evals/vitest.config.ts`.
- A targeted package build was attempted, but it failed on unresolved workspace package imports/types in the isolated build path, not on the `vitest-evals` migration itself.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.