[Bug] assert tp % num_key_value_heads == 0 AssertionError when using Qwen2.5-VL-7B-Instruct-AWQ
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Describe the bug
```
2025-04-03 17:34:37,717 - lmdeploy - WARNING - archs.py:51 - Fallback to pytorch engine because `Qwen/Qwen2.5-VL-7B-Instruct-AWQ/` not supported by turbomind engine.
INFO:qwen_vl_utils.vision_process:set VIDEO_TOTAL_PIXELS: 90316800
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
2025-04-03 17:34:43,333 - lmdeploy - WARNING - transformers.py:22 - LMDeploy requires transformers version: [4.33.0 ~ 4.46.1], but found version: 4.50.2
Traceback (most recent call last):
File "/home/work/workspace/video_vlm/main.py", line 19, in
model_pip = pipeline(model, backend_config=backend_config)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/api.py", line 83, in pipeline
return pipeline_class(model_path,
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/serve/vl_async_engine.py", line 32, in __init__
super().__init__(model_path, backend=backend, backend_config=backend_config, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/serve/async_engine.py", line 279, in __init__
self._build_pytorch(model_path=model_path, backend_config=backend_config, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/serve/async_engine.py", line 341, in _build_pytorch
self.engine = Engine(model_path=model_path, tokenizer=self.tokenizer, engine_config=backend_config)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/engine/engine.py", line 138, in __init__
self.executor = build_executor(model_path,
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/engine/executor/__init__.py", line 70, in build_executor
model_config = ModelConfig.from_pretrained(model_path, trust_remote_code=True, dtype=dtype, tp=tp)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/config.py", line 138, in from_pretrained
return cls.from_hf_config(hf_config, pretrained_model_name_or_path, dtype=dtype, tp=tp)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/config.py", line 145, in from_hf_config
model_config = AutoModelConfigBuilder.build(hf_config, model_path, tp=tp)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/configurations/builder.py", line 47, in build
cfg = valid_builder.build(hf_config, model_path, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/configurations/default.py", line 26, in build
num_key_value_heads = cls.update_num_kv_heads(hf_config, tp, num_key_value_heads)
File "/usr/local/lib/python3.10/dist-packages/lmdeploy/pytorch/configurations/builder.py", line 58, in update_num_kv_heads
assert tp % num_key_value_heads == 0
AssertionError
```
### Reproduction
```
model = 'Qwen/Qwen2.5-VL-7B-Instruct-AWQ/'
backend_config = TurbomindEngineConfig(model_format='awq',tp=8)
model_pip = pipeline(model, backend_config=backend_config)
```
**if i change tp=4 or using 72B model and set tp=8 ; The problem did not occur again.**
### Environment
```Shell
sys.platform: linux
Python: 3.10.11 (main, Apr 5 2023, 14:15:10) [GCC 9.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-40GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.66
GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 2.4.0+cu121
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.1
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 90.1 (built against CUDA 12.4)
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,
TorchVision: 0.19.0+cu121
LMDeploy: 0.7.2.post1+35594d5
transformers: 4.50.2
gradio: Not Found
fastapi: 0.115.12
pydantic: 2.11.0
triton: 3.0.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 CPU Affinity NUMA Affinity
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 PXB SYS SYS 0-19,80-99 0
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 PXB SYS SYS 0-19,80-99 0
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS NODE PXB 20-39,100-119 1
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS NODE PXB 20-39,100-119 1
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS SYS SYS 40-59,120-139 2
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS SYS SYS 40-59,120-139 2
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS SYS SYS 60-79,140-159 3
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS SYS SYS 60-79,140-159 3
NIC0 PXB PXB SYS SYS SYS SYS SYS SYS X SYS SYS
NIC1 SYS SYS NODE NODE SYS SYS SYS SYS SYS X NODE
NIC2 SYS SYS PXB PXB SYS SYS SYS SYS SYS NODE X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
```
### Error traceback
```Shell
```
Contributor guide
Assessment
This issue has not been assessed yet.