mattpocock / mattpocock/evalite
Feature Request (CLI) - Show Failing Scorers
Open
Nobody has claimed this yet.
triaged
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Would love to be able to see which scorers have failed from the CLI. Particularly for use in CI.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the CLI path that reports evaluation results and the scorer status data it exposes. Define the expected output for failed scorers in CI and add coverage for that CLI behavior; done when failed scorer names are visible from the CLI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100