huggingface / huggingface/alignment-handbook

Using MT-Bench to evaluate zephyr

Open
#114 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
490
Avg merge
2m
Merged PRs (30d)
1

Description

In the Readme.md [here](https://github.com/huggingface/alignment-handbook/tree/main/scripts#evaluating-chat-models), it says :

- `Make sure the word zephyr exists in the --model-path argument when generating the model responses...`

We should also ensure the word zephyr exists in the `--model-id` argument

This is because:
1. in the MT-Bench code, they seem to be passing `model_id` around. [code ](https://github.com/lm-sys/FastChat/blob/9924687b67d62032641640bd245d682c4d2f025e/fastchat/llm_judge/gen_model_answer.py#L107)
2. They look for word "zephyr" to find the matching adapter . [here](https://github.com/lm-sys/FastChat/blob/9924687b67d62032641640bd245d682c4d2f025e/fastchat/model/model_adapter.py#L1987)

This is probably a bug in `FastChat`.

Nonetheless we should update the README.md here too. Otherwise, people using the `alignment-handbook` will see low scores on MT-Bench

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.