mattpocock/evalite

View on GitHub

Evaluate your LLM-powered apps with TypeScript

Stars: 1.7k
Forks: 102
Open beginner issues: 0
Indexed issues: 47

Dominant language: TypeScript
License: MIT
Last GitHub push: Nov 10, 2025
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

47 open issues indexed

Is this still an active project Open

Difficulty 5/5 Over a week Newbie friendliness 10/100

mattpocock/evalite#405 · 13 reactions · Opened Aug 13, 2026
UI fails on any remote access: bundle hardcodes localhost + server has no configurable bind host Open

Difficulty 3/5 1-2 days Newbie friendliness 65/100

mattpocock/evalite#404 · Opened Aug 6, 2026
Support for AI SDK v7 in peerDependencies Open

Difficulty 1/5 Under an hour Newbie friendliness 76/100

mattpocock/evalite#400 · 7 reactions · Opened Jun 29, 2026
phantom dependency from evalite to vitest Open

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

mattpocock/evalite#398 · 1 comment · Opened Jun 9, 2026
Bug: `wrapAISDKModel({ caching: false })` is silently ignored when tracing is enabled Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

mattpocock/evalite#395 · Opened May 7, 2026
Various test suite UI improvements Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mattpocock/evalite#394 · Opened May 5, 2026
Don't log task results when using `evalite serve` Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

mattpocock/evalite#393 · Opened May 5, 2026
Clicking a trace row in the eval detail view does nothing (UI stays on eval-level panel) Open

Difficulty 1/5 1-3 hours Newbie friendliness 82/100

mattpocock/evalite#391 · Opened Apr 21, 2026
possible regression: cannot use `@nuxt/test-utils` `defineVitestProject` directly in evalite.config.ts Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

mattpocock/evalite#390 · Opened Apr 18, 2026
Upgrade to tsgo Open

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

mattpocock/evalite#386 · 4 comments · Opened Mar 26, 2026
`wrapAISDKModel` for `EmbeddingModelV2` Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

mattpocock/evalite#374 · Opened Jan 29, 2026
Vercel's default gateway model doesn't work with built-in LLM scorers Open

triaged

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mattpocock/evalite#372 · 1 comment · 1 reaction · Opened Jan 14, 2026
`evalite export` does not fail if threshold is not reached Open

Difficulty 3/5 1-2 days Newbie friendliness 52/100

mattpocock/evalite#370 · 2 comments · Opened Jan 8, 2026
Add support for running multiple eval files by name Open

enhancement

mattpocock/evalite#367 · 4 reactions · 1 assignee · Opened Dec 16, 2025
Add describe/it or other ways to better organize evals within a file Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mattpocock/evalite#364 · 6 reactions · Opened Dec 9, 2025
wrapAISDKModel: Unsupported content type: reasoning (AI SDK v5 reasoning parts) Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

mattpocock/evalite#361 · 4 comments · 5 reactions · Opened Dec 5, 2025
A/B testing pass "variant" as argument to test data function Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mattpocock/evalite#359 · 1 comment · Opened Nov 30, 2025
Add caveat to 'built-in scorers' that they be used sparingly Open

documentation enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

mattpocock/evalite#356 · 2 comments · 1 reaction · Opened Nov 28, 2025
Bug: Evalite Creates Cache Directory Regardless of `cacheEnabled` Setting Open

mattpocock/evalite#354 · 2 comments · 1 assignee · Opened Nov 27, 2025
Display more trace details Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mattpocock/evalite#353 · 1 reaction · Opened Nov 27, 2025
Score calculation needs more explanation Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

mattpocock/evalite#348 · 1 comment · Opened Nov 21, 2025
Threshold as evalite config param Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 48/100

mattpocock/evalite#347 · 2 reactions · Opened Nov 21, 2025
UI: Show progress and evals as they complete when trialCount > 1 Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 48/100

mattpocock/evalite#345 · 1 comment · Opened Nov 20, 2025
Support `.only` line in `vitest` do focus on specific files/tests Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 30/100

mattpocock/evalite#344 · 1 comment · 1 reaction · Opened Nov 19, 2025
Global Statistics Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mattpocock/evalite#343 · 1 comment · 1 reaction · Opened Nov 18, 2025
Document how to evaluate MCP servers Open

documentation enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mattpocock/evalite#337 · 4 comments · Opened Nov 16, 2025
Command to start the UI without running evals Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 42/100

mattpocock/evalite#335 · 2 comments · Opened Nov 15, 2025
[future] AI SDK v6 Integration Plans Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mattpocock/evalite#334 · 1 reaction · Opened Nov 15, 2025
Clarify difference between eval-driven development and test-driven development in docs Open

documentation enhancement

Difficulty 1/5 Under an hour Newbie friendliness 55/100

mattpocock/evalite#315 · 1 reaction · Opened Nov 10, 2025
Add an "Evalite By Example" section to "What Is Evalite?" Open

documentation enhancement

Difficulty 2/5 Half a day Newbie friendliness 55/100

mattpocock/evalite#313 · Opened Nov 10, 2025
evalite.report() for the simplest-possible integration story Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mattpocock/evalite#308 · 1 reaction · Opened Nov 8, 2025
Add 'copy' button to each page of the docs Open

enhancement

mattpocock/evalite#307 · 6 comments · 1 assignee · Opened Nov 7, 2025
Feature: Allow scorer functions to return arrays of scores Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

mattpocock/evalite#304 · 1 comment · Opened Nov 6, 2025
Ability to hide score columns in UI Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mattpocock/evalite#298 · 2 reactions · Opened Nov 5, 2025
Allow passing the same filtering options from the interactive CLI as arguments/config Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mattpocock/evalite#294 · 2 comments · 1 reaction · Opened Nov 4, 2025
Synthetic Test Data Set Generation Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mattpocock/evalite#292 · 2 comments · Opened Nov 4, 2025
Weighted scorers Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mattpocock/evalite#289 · 4 comments · Opened Nov 2, 2025
add a contributors images in Readme.md Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

mattpocock/evalite#264 · 3 comments · Opened Oct 24, 2025
Git integration: track commit & branch in eval runs Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mattpocock/evalite#236 · 1 reaction · Opened Oct 19, 2025
Eval duration not being tracked properly Open

mattpocock/evalite#223 · 5 comments · 1 assignee · Opened Oct 17, 2025
Adapter: Implement Postgres adapter Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mattpocock/evalite#222 · 2 comments · Opened Oct 17, 2025
Adapter: Implement JSON adapter Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

mattpocock/evalite#221 · Opened Oct 17, 2025
Add Screenshots to Docs Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

mattpocock/evalite#216 · Opened Oct 16, 2025
Feature Request (CLI) - Show Failing Scorers Open

triaged

Difficulty 4/5 3-5 days Newbie friendliness 42/100

mattpocock/evalite#125 · 5 comments · 1 reaction · Opened Mar 31, 2025
Track latency and allow passing to scorers and graphs Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mattpocock/evalite#111 · 1 reaction · Opened Feb 7, 2025
Add Human Evaluation from inside Evalite Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

mattpocock/evalite#92 · 1 comment · 4 reactions · Opened Jan 2, 2025
Handle images and files traced from AI SDK Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mattpocock/evalite#86 · 1 reaction · Opened Dec 31, 2024

mattpocock/evalite

Get new issues in your inbox