mattpocock / mattpocock/evalite

Git integration: track commit & branch in eval runs

Open
#236 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
1.7k
Forks
102
PR merge metrics
No merged PRs in 30d

Description

Problem

Currently evalite doesn't track which git commit or branch an eval was run against. This makes it hard to:

  • Correlate eval results with specific code versions
  • Track performance across commits/branches
  • Understand which code changes affected eval scores

Proposal

Store in eval run metadata:

  • Commit SHA
  • Branch name
  • (Optional) dirty state flag

Implementation considerations

  • Read git info at run start (similar to how we get cwd)
  • Store in runs table (add commit & branch columns)
  • Display in UI (run history, comparison views)
  • Handle non-git repos gracefully
  • Handle dirty working trees

Scope

Likely post-v1 - fairly tasty chunk of work that could expose funny bugs.

Questions

  • Should we block runs if working tree is dirty?
  • Should we auto-compare across branches?
  • Git info in filesystem mode vs server mode?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating where run metadata is collected and persisted in the runs table, then trace the run history and comparison views. Done means recording commit and branch information, handling dirty and non-Git repositories gracefully, and displaying the metadata in the relevant UI.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, typescript
Domain
databases, devtools, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.