evals: wire successPredicates through suite-run hydration (Convex testCase schema + startSuiteRunWithRecorder)
- Dominant language
- TypeScript
- Stars
- 2.2k
- Forks
- 283
- Avg merge
- 11h 47m
- Merged PRs (30d)
- 737
Description
## Summary
Suite-path delivery of `successPredicates` on `EvalTestCase` is intentionally deferred from PR #2352 (https://github.com/MCPJam/inspector/pull/2352#discussion_r3331442419).
Quick-run and single-case paths already gate on `successPredicates` today via `RunTestCaseRequest.testCaseOverrides.successPredicates`. Suite runs cannot yet, because:
1. The Convex `testCase` schema does not include the field.
2. `startSuiteRunWithRecorder` rebuilds `config.tests` from persisted Convex `testCase` rows, so adding the field to `RunEvalsRequestSchema` alone would be accepted-but-silently-ignored.
## Work required
- **Backend (mcpjam-backend, branch `eval-predicate-metadata`)**: Add `successPredicates` to the Convex `testCase` schema and update storage/retrieval accordingly.
- **Inspector**: In `startSuiteRunWithRecorder` / `createTestCase` (~L703-720 of `mcpjam-inspector/server/services/evals-runner.ts`), thread `tc.successPredicates` into the rebuilt `EvalTestCase` objects, mirroring how `matchOptions` is already handled.
- **Schema**: Add `successPredicates` to `RunEvalsRequestSchema` and ensure it survives the round-trip through Convex.
## References
- PR: https://github.com/MCPJam/inspector/pull/2352
- Comment: https://github.com/MCPJam/inspector/pull/2352#discussion_r3331442419
- Companion backend branch: `eval-predicate-metadata`
/cc @chelojimenez
Contributor guide
Assessment
This issue has not been assessed yet.