mattpocock / mattpocock/evalite
`evalite export` does not fail if threshold is not reached
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
When running evalite with export to generate a report that I can upload in CI, I noticed that using a threshold doesn't work as I would expect it. The threshold is correctly outputted as failed, but the command does not fail with exit code. Therefore I can't detect in the CI that something is off. It works without using export, but I would like to have both so that I have something to analyze if it fails.
As this is a breaking change, is that something that should go into v1?
PS: I'm using 1.0.0-beta.15
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with evalite export and a threshold, then trace the CLI path that handles threshold results and export. Done means the command returns a failing exit code when the threshold is not reached while still generating the report; compare this with the existing non-export behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100