Please help add SeaLLM-7B-v2 with 7.54 on MT-bench on the leaderboard
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi Lmsys team,
We released [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) last week - a multilingual model that achieves **7.54** on the English MT-bench.
**Can you please check and verify the results and add our model to the leaderboard!**
## Generation and GPT-4 ratings files:
[https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/tree/main/evaluation/mt_bench](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/tree/main/evaluation/mt_bench)
Please find the files in the link above.
## Steps to reproduce the results:
1. Add the format to [conversations.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py)
```python
# Seallm
register_conv_template(
Conversation(
name="seallm",
system_template="""<|im_start|>system
{system_message}""",
system_message="""You are a helpful, intelligent and safe assistant.""",
roles=("<|im_start|>user", "<|im_start|>assistant"),
sep_style=SeparatorStyle.CHATML,
sep="",
stop_token_ids=[0, 1],
)
)
```
2. Add the following to [model_adapter.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/model_adapter.py)
```python
class SeaLLMAdapter(BaseModelAdapter):
"""The model adapter for SeaLLMAdapter"""
use_fast_tokenizer = False
def match(self, model_path: str):
return "sea" in model_path.lower()
def get_default_conv_template(self, model_path: str) -> Conversation:
return get_conv_template("seallm")
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fastchat/conversation.py and fastchat/model/model_adapter.py, following the SeaLLM conversation template and adapter shown in the issue. Review the linked SeaLLM generation and GPT-4 ratings files and the existing leaderboard evaluation flow. Done means SeaLLM-7B-v2 is supported and its reported MT-bench result has been checked for leaderboard inclusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100