Minor typos in judge prompts
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
In llm_judge/data/judge_prompts.jsonl there are two "You evaluation" typos which should be "Your evaluation"
- "You evaluation should focus on the assistant's answer to the second question" in "single-math-v1-multi-turn"
- "You evaluation should focus on the assistant's answer to the second question" in "single-v1-multi-turn"
Doubt they make any difference to performance as this is an easy error for models to parse but worth correcting for completeness.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Open fastchat/llm_judge/data/judge_prompts.jsonl and locate the single-math-v1-multi-turn and single-v1-multi-turn entries. Check both occurrences of "You evaluation" and confirm they read "Your evaluation" when the two prompt strings are corrected.
Written by the indexing model from the issue text.
Assessment
- Domain
- content
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100