huggingface / huggingface/alignment-handbook
Using MT-Bench to evaluate zephyr
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 490
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
In the Readme.md [here](https://github.com/huggingface/alignment-handbook/tree/main/scripts#evaluating-chat-models), it says :
- `Make sure the word zephyr exists in the --model-path argument when generating the model responses...`
We should also ensure the word zephyr exists in the `--model-id` argument
This is because:
1. in the MT-Bench code, they seem to be passing `model_id` around. [code ](https://github.com/lm-sys/FastChat/blob/9924687b67d62032641640bd245d682c4d2f025e/fastchat/llm_judge/gen_model_answer.py#L107)
2. They look for word "zephyr" to find the matching adapter . [here](https://github.com/lm-sys/FastChat/blob/9924687b67d62032641640bd245d682c4d2f025e/fastchat/model/model_adapter.py#L1987)
This is probably a bug in `FastChat`.
Nonetheless we should update the README.md here too. Otherwise, people using the `alignment-handbook` will see low scores on MT-Bench
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.