when inferencing the deepspeed saved model, the output is messy

I have revised the model_adapter.py and the conversation.py.

In model_adapter.py:

class M30Adapter(BaseModelAdapter):
    def match(self, model_path: str):
        return "mistral-30k" in model_path
    def load_model(self, model_path: str, from_pretrained_kwargs: dict):
        tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
        model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **from_pretrained_kwargs)
    def get_default_conv_template(self, model_path: str) -> Conversation:
        return get_conv_template("mistral-30k")
...
register_model_adapter(M30Adapter)

In conversation.py,

register_conv_template(
    Conversation(
        name="mistral-30k",
        system_message="A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n",
        roles=("Human", "Assistant"),
        sep_style=SeparatorStyle.PHOENIX,
        sep="</s>",
    )
)

but when I run:

python3 -m fastchat.serve.cli --model-path /balabala/ReMax/step3_rlhf_finetuning/log/step3_remax-mistral-30k-2024-02-28-16-31-11-2023/actor --debug

the output is messy:

{'conv_template': 'mistral-30k', 'prompt': "A chat between a curious human and an artificial intelligen$
e assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\nHum$
n: <s>hi</s>Assistant: <s>", 'outputs': 'It term isolministrном boolean tossedций fractapolis Wonderlayo
utizont String hesitatedTIESobjectsMPberry cris\');Manifestpfnego eg put stoparenENCsecurity hadeany dev
astiledmsguent srcющи Produ academpack Aprգ refres Further few�acyj surroundingToolsamd dickкомGenerateM fetyacion bit Apple terminviously MPrivingologieHT Hausд firmly stud▸     aspir respectively chan[0/673]
s Kyle federalる Group HarborResponse Storage hotels)^constSSL moraletz migrations Didtracklang($markers
gpingkal accidentallyтелulent=Propisine version concent HoldFunimetsix Experience SClua casehorizontal O
ther WHE Milintonovenzas__( hirediago Character rapidly successfulheart fetchNeedрат杂 Useulse guyOwner#
{delelinesortsiding ге Jah operates nomin cntestly Con saddleよ prop barn click values optimal prove off
er Lin Bes ancestorsuni mechanism aristIdx withش consciencewick workspace>= exponent', 'speed (token/s)'
: 37.29}

The actor directory consists of three files:

(remaxx) [kongchuyi@pgpu18 actor]$ ls -lh
total 14G
-rw-r--r-- 1  733 Feb 28 21:14 config.json
-rw-r--r-- 1  14G Feb 28 21:14 pytorch_model.bin
-rw-r--r-- 1  482K Feb 28 21:14 tokenizer.model

Reproduce the reported command in fastchat.serve.cli with the actor directory, then inspect model_adapter.py and conversation.py, including the M30Adapter and mistral-30k conversation template. Check the three files in the actor directory and compare the generated output with the intended inference behavior; done means the model no longer produces the reported messy output.