MCPJam / MCPJam/inspector

evals: wire successPredicates through suite-run hydration (Convex testCase schema + startSuiteRunWithRecorder)

Open
#2,353 0 comments 0 reactions 1 assignee Claimed by @chelojimenez View on GitHub
enhancement
Dominant language
TypeScript
Stars
2.2k
Forks
283
Avg merge
11h 47m
Merged PRs (30d)
737

Description

## Summary

Suite-path delivery of `successPredicates` on `EvalTestCase` is intentionally deferred from PR #2352 (https://github.com/MCPJam/inspector/pull/2352#discussion_r3331442419).

Quick-run and single-case paths already gate on `successPredicates` today via `RunTestCaseRequest.testCaseOverrides.successPredicates`. Suite runs cannot yet, because:

1. The Convex `testCase` schema does not include the field.
2. `startSuiteRunWithRecorder` rebuilds `config.tests` from persisted Convex `testCase` rows, so adding the field to `RunEvalsRequestSchema` alone would be accepted-but-silently-ignored.

## Work required

- **Backend (mcpjam-backend, branch `eval-predicate-metadata`)**: Add `successPredicates` to the Convex `testCase` schema and update storage/retrieval accordingly.
- **Inspector**: In `startSuiteRunWithRecorder` / `createTestCase` (~L703-720 of `mcpjam-inspector/server/services/evals-runner.ts`), thread `tc.successPredicates` into the rebuilt `EvalTestCase` objects, mirroring how `matchOptions` is already handled.
- **Schema**: Add `successPredicates` to `RunEvalsRequestSchema` and ensure it survives the round-trip through Convex.

## References

- PR: https://github.com/MCPJam/inspector/pull/2352
- Comment: https://github.com/MCPJam/inspector/pull/2352#discussion_r3331442419
- Companion backend branch: `eval-predicate-metadata`

/cc @chelojimenez

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.