How to generate reference answers in MT-Bench?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
Thank you for your excellent work on the LLM evaluation! I'm inspired to create a French version of MT-Bench.
Currently, I'm in the process of generating reference answers for tasks in the math, reasoning, and coding categories. I would appreciate more details on your approach, as I've found myself a bit confused about which version of GPT-4 to use.
At the moment, I'm using `gpt-4-0613` as the judge and am looking to evaluate `gpt-4-0314` later. My understanding is that I can't use `gpt-4-0314` to generate reference answers since the candidate answers from `gpt-4-0314` will be identical, leading to a consistent 10/10 score on all three tasks.
I've considered using `gpt-4-1106-preview`, but this also means I can't evaluate `gpt-4-1106-preview` later. I would like to learn how you've dealt with similar cases and which version you would recommend. Thanks in advance!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no repository files, tests, or entry points; start by tracing the MT-Bench reference-answer and judge workflow described in the issue. Done means documenting the recommended GPT-4 versioning approach for French MT-Bench reference answers and how to avoid evaluating a model against its own outputs.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100