A100 GPU resource utilization is also not high.

Hi there,

I have deployed the Vicuna-13B model successfully on my NVIDIA A100 GPU. However, I am experiencing slow token generation speeds and low GPU resource utilization. As you can see in the example below, despite opening four Gradio web chat instances, the A100 GPU utilization rate remains at only 26%, and the token generation speed is significantly slow.

I am seeking advice on how to optimize resource utilization to improve the performance of the model.

The deploy command used is outlined below:

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --host 127.0.0.1 --port 31000 --worker-address=http://127.0.0.1:31000 --controller-address http://localhost:21001 --model-name "Vicuna-13b-v1.1" --model-path models/vicuna-13b --gpus 0
python3 -m fastchat.serve.gradio_web_server --model-list-mode reload

Thank you.

Start with the deployment entry points in fastchat.serve.controller, fastchat.serve.model_worker, and fastchat.serve.gradio_web_server, using the reported Vicuna-13B command and four concurrent Gradio sessions as the reproduction case. Compare GPU utilization and token generation speed during serving; done means identifying a confirmed bottleneck and documenting an actionable optimization.