fastchat.serve.model_worker not loading checkpoint shards and outputs stderr messages

Nobody has claimed this yet.

Dear FastChat Developers,

I am part of a research group working with integrating Semantic Web-based Knowledge Graphs and LLMs such as Vicuna. We are working on an open-source research project, in order to submit a paper to a Semantic Web conference.

I followed all your guidelines w.r.t. setting up Vicuna weights as delta weights from Llama.

So far, I have been able to run under Windows 11 on a 64GB RAM/4GB GPU machine, the Inference with Command Line Interface section by running: python -m fastchat.serve.cli --model-path ./models/vicuna-13b --device cpu --load-8bit

It runs OK but really slow. I assume with a machine with several GPUs or a powerful single GPU can run the python scripts properly and loads the Checkpoint Shards without any problems.

My interest is in running the restful API and client SDK because I want to submit a large set of prompts and store the answers. When I run the following command:

python -m fastchat.serve.controller this is the screen that appears:

And when I run: python -m fastchat.serve.model_worker --model-name 'vicuna-13b' --model-path ./models/vicuna-13b --device cpu --load-8bit

It doesn't load the checkpoint shards and starts outputting stderr errors. This is the screen where you can notice that on the previous command, it loaded properly the checkpoint shards but here it doesn't. Instead it starts outputting a series of stderr messages.


The is an error line coming from fastchat/utils line 86 def write(self, buf). I assume it must be a small error.

Any hints on how to overcome this and execute properly for submitting in batches prompts to the Vicuna model via the client python SDK API?

Any help would be highly regarded and we will include a reference to FastChat in our research paper. Thanks.

Regards, Carlos F. Enguix

Start with the fastchat.serve.model_worker command using the reported Vicuna path and options, and compare its output with fastchat.serve.cli. Then inspect fastchat/utils.py around line 86 and capture the complete stderr output; done means the model worker loads the checkpoint shards and serves batch prompts through the client API without those messages.

Written by the indexing model from the issue text.