mattpocock / mattpocock/evalite
Add Human Evaluation from inside Evalite
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
The idea is:
- You would specify a special
humanScorer()in thescoresarray passed toevalite() - In the UI, you could mark the result with a tick or cross
- This would be saved to a local JSON file, keyed under the eval name, input, expected and output. (this means that if the input, expected or output changes, the human score would be invalidated)
- The UI would then retrieve this JSON information where needed to display the scores.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the evalite() scores array and the UI flow described in the issue, then trace how scores are stored and displayed. Define the humanScorer() interface and local JSON keying before connecting the UI tick or cross to persistence. Done means human marks are shown and retained, and changing the eval name, input, expected value, or output invalidates the saved mark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100