mattpocock / mattpocock/evalite
Git integration: track commit & branch in eval runs
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Problem
Currently evalite doesn't track which git commit or branch an eval was run against. This makes it hard to:
- Correlate eval results with specific code versions
- Track performance across commits/branches
- Understand which code changes affected eval scores
Proposal
Store in eval run metadata:
- Commit SHA
- Branch name
- (Optional) dirty state flag
Implementation considerations
- Read git info at run start (similar to how we get
cwd) - Store in
runstable (addcommit&branchcolumns) - Display in UI (run history, comparison views)
- Handle non-git repos gracefully
- Handle dirty working trees
Scope
Likely post-v1 - fairly tasty chunk of work that could expose funny bugs.
Questions
- Should we block runs if working tree is dirty?
- Should we auto-compare across branches?
- Git info in filesystem mode vs server mode?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating where run metadata is collected and persisted in the runs table, then trace the run history and comparison views. Done means recording commit and branch information, handling dirty and non-Git repositories gracefully, and displaying the metadata in the relevant UI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, typescript
- Domain
- databases, devtools, full-stack
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100