lm-sys / lm-sys/FastChat

vicuna-7b fastchat.serve.cli stops loading checkpoint shards in my google colab

Open
#1,771 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

I've been trying to get fastchat to work many times but i always get stuck here. I'm using a google colab with python 3.10.12 and i've followed the installation instructions.
i'm trying to run this:
!python -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3

and here's the output:
023-06-24 16:34:14.704392: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Loading checkpoint shards: 50% 1/2 [01:23<01:23, 83.38s/it]^C

(i did not stop it)
not sure if it has to deal with trensorRT.
I've already tried with vicuna-7b-v1.3, vicuna-13b-v1.3 and fastchat-t5-3b-v1.0 and they all end up with this output.
I've also tried with only using cpu and still the same.

Thanks

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported command, python -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3, in the stated Google Colab and Python 3.10.12 environment. Start from the fastchat.serve.cli entry point and determine whether the TensorRT warning or checkpoint loading is responsible; done means identifying the cause and confirming that the model loads successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.