mattpocock / mattpocock/evalite
Eval duration not being tracked properly
Open
@erayack is already working on this.
Since Oct 20, 2025.
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The eval duration field in the database is always 0. Test failure:
FAIL tests/basics.test.ts > Should capture the duration as being more than 0
AssertionError: expected 0 to be greater than 0
The evals table has a duration column, but it's never being populated with actual timing data.
Root Cause
In the adapter system refactor, eval-level duration tracking was lost:
updateEvalStatusAndDuration()indb.ts:626-650only updatesstatus, notdurationEvalite.Adapter.Evals.UpdateOptsintypes.ts:300-303only hasidandstatus- Reporter has
onTestModuleStart/onTestModuleEndhooks but doesn't track/report timing
Proposed Solution
Track eval timing in reporter:
- Add
duration?: numbertoEvalite.Adapter.Evals.UpdateOpts - In
EvaliteRunner: Track module start times in Map onTestModuleStart: Storeperformance.now()for moduleonTestModuleEnd: Calculate duration, pass to runner- Runner passes duration to
adapter.evals.update() - Update
updateEvalStatusAndDuration()to accept/set duration
Flow:
Module start → store timestamp → module end → calc duration → update eval with duration + status
Unresolved Questions
-
What should eval duration represent?
- Total wall-clock time from module start to end?
- Sum of individual test durations?
- Time excluding setup/teardown?
-
Do we even want to track eval duration?
- Is this metric useful to users?
- Result-level duration already tracked
- Does eval duration add value?
-
Alternative: Skip or remove the feature?
- Could remove
durationcolumn from evals table - Focus on result-level timing only
- Simplify data model
- Could remove
Temporary Fix
Skip failing test until decision made on tracking approach.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.