When using the new version of 'mtbench', I encountered an issue
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
When I was using the latest version of the code and running mt-bench within llm-judge, I encountered the following error:
```bash
llm_judge git:(main) ✗ python3 gen_model_answer.py --model-path /chenhui/zhangwuhan/model/baichuan2 --model-id baichuan2-base
[2023-09-20 15:49:04,731] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Output to data/mt_bench/model_answer/baichuan2-base.jsonl
Traceback (most recent call last):
File "/chenhui/zhangwuhan/evaluation/FastChat/fastchat/llm_judge/gen_model_answer.py", line 267, in
run_eval(
File "/chenhui/zhangwuhan/evaluation/FastChat/fastchat/llm_judge/gen_model_answer.py", line 54, in run_eval
get_answers_func(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/chenhui/zhangwuhan/evaluation/FastChat/fastchat/llm_judge/gen_model_answer.py", line 83, in get_model_answers
model, tokenizer = load_model(
TypeError: load_model() got an unexpected keyword argument 'dtype'
```
I removed the 'dtype' parameter from the 'load_model' function. Will this affect the results? Also, will the results from the new version of 'mt-bench' differ significantly from the old version?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fastchat/llm_judge/gen_model_answer.py, especially run_eval and get_model_answers, and inspect the load_model definition referenced by the traceback. Reproduce the shown mt-bench command, then verify that the call and function signature agree and determine whether removing dtype affects evaluation results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100