OpenEuroLLM / OpenEuroLLM/Taskboard
Rubrics for LLM judges
Open
@kargibora is already working on this.
Since Mar 3, 2026.
- Dominant language
- No language data
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Currently, we use OpenJury for LLM judges which supports only pairwise evaluations for single criteria.
We want to support also rubrics to score LLMs in several axis (helpfulness, style, ...) so that scores are more interpretable.
We want to:
- add support to evaluate rubrics in OpenJury
- add a report when evaluating with rubrics which allows to interpret the different strengths / weaknesses of the model
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.