mattpocock / mattpocock/evalite
Feature: Allow scorer functions to return arrays of scores
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Feature Request: Allow scorer functions to return arrays of scores
Problem
Currently, scorer functions can only return a single Score object. However, some use cases involve LLM graders that assess multiple criteria in a single call and return multiple pass/fail results.
Use Case
I have an LLM grader that performs a one-shot assessment of multiple criteria and returns a pass/fail for each criterion. For example, evaluating if a response is accurate, complete, and clear - all in a single LLM call for efficiency.
Currently, I need to either:
- Make separate LLM calls for each criterion (inefficient and costly)
- Write custom logic to split the results into multiple scorer calls
- Use only a single aggregate score
Proposed Solution
Allow scorers to return either a single Score or an array of Score[], which would be flattened into the final scores array.
This appears to be achievable with a simple .flat() here: https://github.com/mattpocock/evalite/blob/e04b48ff260d8966e1fa3b2a6955c9abae9adcb9/packages/evalite/src/evalite.ts#L89 (plus type changes to allow Score | Score[]).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/evalite/src/evalite.ts at the linked location around line 89, then trace the scorer result types. Update the accepted result shape and flatten returned score arrays into the final scores array; done means both single Score results and arrays of Score results are supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100