mattpocock / mattpocock/evalite

Eval duration not being tracked properly

Open
#223 5 comments 0 reactions 1 assignee View on GitHub

@erayack is already working on this.

Since Oct 20, 2025.

Dominant language
TypeScript
Stars
1.7k
Forks
102
PR merge metrics
No merged PRs in 30d

Description

Problem

The eval duration field in the database is always 0. Test failure:

FAIL  tests/basics.test.ts > Should capture the duration as being more than 0
AssertionError: expected 0 to be greater than 0

The evals table has a duration column, but it's never being populated with actual timing data.

Root Cause

In the adapter system refactor, eval-level duration tracking was lost:

  1. updateEvalStatusAndDuration() in db.ts:626-650 only updates status, not duration
  2. Evalite.Adapter.Evals.UpdateOpts in types.ts:300-303 only has id and status
  3. Reporter has onTestModuleStart/onTestModuleEnd hooks but doesn't track/report timing

Proposed Solution

Track eval timing in reporter:

  1. Add duration?: number to Evalite.Adapter.Evals.UpdateOpts
  2. In EvaliteRunner: Track module start times in Map
  3. onTestModuleStart: Store performance.now() for module
  4. onTestModuleEnd: Calculate duration, pass to runner
  5. Runner passes duration to adapter.evals.update()
  6. Update updateEvalStatusAndDuration() to accept/set duration

Flow:
Module start → store timestamp → module end → calc duration → update eval with duration + status

Unresolved Questions

  1. What should eval duration represent?

    • Total wall-clock time from module start to end?
    • Sum of individual test durations?
    • Time excluding setup/teardown?
  2. Do we even want to track eval duration?

    • Is this metric useful to users?
    • Result-level duration already tracked
    • Does eval duration add value?
  3. Alternative: Skip or remove the feature?

    • Could remove duration column from evals table
    • Focus on result-level timing only
    • Simplify data model

Temporary Fix

Skip failing test until decision made on tracking approach.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.