InternLM / InternLM/lmdeploy

[Bug] qwen1.5-14b-awq服务部署error

Open
#1,532 5 comments 0 reactions 1 assignee Claimed by @lvhan028 View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.

### Describe the bug

使用lmdeploy lite auto_awq生成awq模型;
然后使用tp=2部署服务,会出现如下error:

2024-04-30 08:25:44,179 - lmdeploy - INFO - input backend=turbomind, backend_config=TurbomindEngineConfig(model_name='qwen', model_format='awq', tp=2, session_len=16384, max_batch_size=4, cache_max_entry_count=0.8, cache_block_seq_len=64, quant_policy=8, rope_scaling_factor=0.0, use_logn_attn=False, download_dir=None, revision=None, max_prefill_token_num=8192, num_tokens_per_iter=0, max_prefill_iters=1)
2024-04-30 08:25:44,179 - lmdeploy - INFO - input chat_template_config=None
2024-04-30 08:25:45,243 - lmdeploy - INFO - updated chat_template_onfig=ChatTemplateConfig(model_name='qwen', system=None, meta_instruction=None, eosys=None, user=None, eoh=None, assistant=None, eoa=None, separator=None, capability=None, stop_words=None)
2024-04-30 08:25:45,243 - lmdeploy - WARNING - model_source: hf_model
2024-04-30 08:25:45,243 - lmdeploy - WARNING - model_name is deprecated in TurbomindEngineConfig and has no effect
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-04-30 08:25:46,965 - lmdeploy - WARNING - model_config:

[llama]
model_name = qwen
tensor_para_size = 2
head_num = 40
kv_head_num = 40
vocab_size = 152064
num_layer = 40
inter_size = 13696
norm_eps = 1e-06
attn_bias = 1
start_id = 151643
end_id = 151645
session_len = 16384
weight_type = int4
rotary_embedding = 128
rope_theta = 1000000.0
size_per_head = 128
group_size = 128
max_batch_size = 4
max_context_token_num = 1
step_length = 1
cache_max_entry_count = 0.8
cache_block_seq_len = 64
cache_chunk_size = -1
num_tokens_per_iter = 8192
max_prefill_iters = 2
extra_tokens_per_iter = 0
use_context_fmha = 1
quant_policy = 8
max_position_embeddings = 32768
rope_scaling_factor = 0.0
use_dynamic_ntk = 0
use_logn_attn = 0
lora_policy =
lora_r = 0
lora_scale = 0.0
lora_max_wo_r = 0
lora_rank_pattern =
lora_scale_pattern =

[TM][INFO] Set logger level by INFO
[TM][WARNING] [LlamaTritonModel] `max_context_token_num` = 16384.
[TM][INFO] Set logger level by INFO
[TM][INFO] Set logger level by INFO
2024-04-30 08:25:47,770 - lmdeploy - WARNING - get 883 model params
Convert to turbomind format: 0%| | 0/40 [00:00
load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')()
File "/opt/lmdeploy/lmdeploy/cli/entrypoint.py", line 37, in run
args.run(args)
File "/opt/lmdeploy/lmdeploy/cli/serve.py", line 283, in api_server
run_api_server(args.model_path,
File "/opt/lmdeploy/lmdeploy/serve/openai/api_server.py", line 1217, in serve
VariableInterface.async_engine = pipeline_class(
File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 206, in __init__
self._build_turbomind(model_path=model_path,
File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 254, in _build_turbomind
self.engine = tm.TurboMind.from_pretrained(
File "/opt/lmdeploy/lmdeploy/turbomind/turbomind.py", line 396, in from_pretrained
return cls(model_path=pretrained_model_name_or_path,
File "/opt/lmdeploy/lmdeploy/turbomind/turbomind.py", line 170, in __init__
self.model_comm = self._from_hf(model_source=model_source,
File "/opt/lmdeploy/lmdeploy/turbomind/turbomind.py", line 305, in _from_hf
output_model.export()
File "/opt/lmdeploy/lmdeploy/turbomind/deploy/target_model/base.py", line 273, in export
self.export_transformer_block(bin, i)
File "/opt/lmdeploy/lmdeploy/turbomind/deploy/target_model/w4.py", line 156, in export_transformer_block
self.save_split(w2_sz, f'layers.{i}.feed_forward.w2.scales_zeros', 0)
File "/opt/lmdeploy/lmdeploy/turbomind/deploy/target_model/base.py", line 246, in save_split
assert tensor.shape[split_dim] % tp == 0
AssertionError

### Reproduction

lmdeploy serve api_server ../pretrained-models/qwen1.5-14b-chat-w4-lmdeploy/ --backend turbomind --model-format awq --log-level INFO --tp 2 --quant-policy 8

### Environment

```Shell
使用docker镜像:0.4.0
```

### Error traceback

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.