lm-sys / lm-sys/FastChat

Failed to import tinychat.

Open
#3,038 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

Installation is fine. The following is my setup. If I run the model_worker, error message with Failed to import tinychat.

PS: I am using the Qwen AWQ model. Qwen 14B AWQ

awq                           0.1.0        /home/user/tmp/FastChat/repositories/llm-awq
awq-inference-engine          0.0.0
fschat                        0.2.36       /home/user/tmp/FastChat
torch                         2.1.2+cu121
torchaudio                    2.1.2+cu121
torchvision                   0.16.2+cu121
python3 -m fastchat.serve.model_worker --model-name Qwen --model-path ./ --num-gpus 3 --awq-wbits 4 --awq-groupsize 128

2024-02-12 22:45:04 | INFO | model_worker | Loading the model ['Qwen'] on worker 2ed2e3b9 ...
2024-02-12 22:45:04 | INFO | stdout | Loading AWQ quantized model...
2024-02-12 22:45:04 | INFO | numexpr.utils | Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-02-12 22:45:04 | INFO | numexpr.utils | NumExpr defaulting to 8 threads.
2024-02-12 22:45:04 | INFO | datasets | PyTorch version 2.1.2+cu121 available.
2024-02-12 22:45:04 | INFO | stdout | Error: Failed to import tinychat. cannot import name 'real_quantize_model_weight' from 'awq.quantize.quantizer' (/home/chaoyou/anaconda3/envs/fastchat/lib/python3.10/site-packages/awq/quantize/quantizer.py)
2024-02-12 22:45:04 | INFO | stdout | Please double check if you have successfully installed AWQ
2024-02-12 22:45:04 | INFO | stdout | See https://github.com/lm-sys/FastChat/blob/main/docs/awq.md

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the fastchat.serve.model_worker entry point and review the AWQ setup described in docs/awq.md. Reproduce the Qwen AWQ command and trace the failed tinychat import; done means the model_worker starts without the reported import error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.