Mixtral-8x7B-Instruct-v0.1: Chat arena vs local inference

Nobody has claimed this yet.

Here how I made Mixtral-8x7B-Instruct-v0.1 work using FastChat.vllm_worker.

- Python3.10 given that megablocks only works with Python3.10
- install vllm 0.2.4 version (newer versions of vLLM are having few issues like [#2219](https://github.com/vllm-project/vllm/issues/2219) , [#2229](https://github.com/vllm-project/vllm/issues/2229))
- checkpoints in .pt format
- cuda version >=12.1
- pip install megablocks

**Still I find the answers from the chat arena's Mixtral-8x7B-Instruct-v0.1 much better.**

```
curl --location 'http://$IP:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
"model": "Mixtral",
"messages": [
{
"role": "user",

"content":"who are you?"

}
],
"temprature": 0.7,
"max_tokens": 1024,
"top_p":1
}'

```

Answer from local inference.

```
{
"id": "chatcmpl-GohBJuu5P6kXkDzU3BsCib",
"object": "chat.completion",
"created": 1703237717,
"model": "Mixtral",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": " I am an artificial intelligence assistant, designed to help answer questions, provide information, and assist with various tasks to make your life easier and more convenient."
},
"finish_reason": "stop"

}
],
"usage": {
"prompt_tokens": 540,
"total_tokens": 571,
"completion_tokens": 31
}
}
```

------
Answer from the Chat Arena.

------
Still have few questions

**_Is there anything that I'm still missing?_**
- I knew LLM are not going to produce the same result but still the answer I'm getting using my local inference are still having lesser quality compared to Arena.
- Is there any specific system prompt I need to use to get the right quality output?
- What is the average throughput of Mixtral-8x7B-Instruct-v0.1 model? Any information around same?

Start with the FastChat.vllm_worker setup described in the issue, including Python 3.10, vLLM 0.2.4, Megablocks, CUDA, and the chat completions curl request. Compare the local Mixtral-8x7B-Instruct-v0.1 configuration and output with the Chat Arena example. Done would require identifying the missing configuration or explaining the quality and throughput difference, but the issue does not define a concrete fix or test.

Written by the indexing model from the issue text.