[Bug] lmdeploy Turbomind 推理导致jupyter 崩溃

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

在vscode上python3.12 使用lmdeploy 推理。导致jupyter内核崩溃。不知道是哪里的问题


### Reproduction

```python
import os
os.environ["LMDEPLOY_USE_MODELSCOPE"] = "True"
from modelscope import snapshot_download
from modelscope.hub.file_download import model_file_download

MODEL_PATH = snapshot_download('qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4')
from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig
pipe = pipeline(MODEL_PATH,
backend_config=TurbomindEngineConfig(
max_batch_size=32,
enable_prefix_caching=True,
cache_max_entry_count=0.8,
session_len=8192,
quant_policy=4,
))
gen_config = GenerationConfig(top_p=0.8,
top_k=40,
temperature=0.8,
do_sample=True,
max_new_tokens=1024)
```
```python
response = pipe(['Hi, pls intro yourself', 'Shanghai is'],
gen_config=gen_config)

print(response)
```

### Environment

```Shell
(base) root@PS2023STCKYAEB:/mnt/d/my/work/study/ai/kaggle_code/aimo2# lmdeploy check_env
sys.platform: linux
Python: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 13:27:36) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr/local/cuda-12.6
NVCC: Cuda compilation tools, release 12.6, V12.6.77
GCC: gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
PyTorch: 2.5.1+cu124
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 12.4
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 90.1
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.20.1+cu124
LMDeploy: 0.7.0.post3+7780d7d
transformers: 4.48.2
gradio: 5.13.1
fastapi: 0.115.4
pydantic: 2.8.2
triton: 3.1.0
NVIDIA Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
```

### Error traceback

```Shell
21:53:05.031 [trace] Start cell 2 execution @ 1739195585031 (clear output)
21:53:05.031 [debug] Kernel acknowledged execution of cell 2 @ 1739195585031
21:53:08.478 [debug] KernelProcess error 54635: [TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 8192.

21:53:15.223 [trace] Executing silently Code (busy) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a
21:53:15.223 [trace] Executing silently Code (busy) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a
21:53:15.895 [debug] Kernel output 54635: [WARNING] gemm_config.in is not found; using default GEMM algo

21:53:28.864 [trace] Cell 2 completed in 23.833s (start: 1739195585031, end: 1739195608864)
21:53:28.864 [trace] Cell 2 executed successfully
21:53:28.864 [trace] Cell Index:3 sent to kernel
21:53:28.881 [trace] Executing silently Code (completed) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a with 1 output(s)
21:53:28.925 [trace] Start cell 3 execution @ 1739195608925 (clear output)
21:53:28.925 [debug] Kernel acknowledged execution of cell 3 @ 1739195608925
21:53:28.925 [trace] Executing silently Code (completed) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a with 1 output(s)
21:53:28.925 [trace] Cell 3 completed in 0s (start: 1739195608925, end: 1739195608925)
21:53:28.926 [trace] Cell 3 executed successfully
21:53:28.926 [trace] Cell Index:0 sent to kernel
21:53:28.930 [trace] Start cell 0 execution @ 1739195608930 (clear output)
21:53:28.930 [debug] Kernel acknowledged execution of cell 0 @ 1739195608930
21:53:28.931 [trace] Cell 0 completed in 0.001s (start: 1739195608930, end: 1739195608931)
21:53:28.931 [trace] Cell 0 executed successfully
21:53:40.511 [debug] Handle Execution of Cells 4 for /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb
21:53:40.511 [trace] Execute Notebook /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb. Step 1
21:53:40.511 [trace] Connect to Kernel /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb. Step 2
21:53:40.511 [trace] Connected to Kernel /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb. Step 3
21:53:40.511 [trace] executeCell 4. Step 4
21:53:40.511 [trace] Cell Index:4 sent to kernel
21:53:40.514 [trace] Start cell 4 execution @ 1739195620514 (clear output)
21:53:40.514 [debug] Kernel acknowledged execution of cell 4 @ 1739195620514
21:53:47.782 [debug] KernelProcess Exited 54635, Exit Code - null [TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 8192.

21:53:47.782 [error] Disposing session as kernel process died ExitCode: undefined, Reason: [TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 8192.

21:53:47.783 [debug] Dispose Kernel process 54635.
21:53:47.787 [debug] Disposed Kernel process 54635.
21:53:47.806 [trace] Cell 4 completed in -1739195620.514s (start: 1739195620514, end: undefined)
```
改成pytorch 推理后端报下列错误。是需要使用awq量化？
```output
RuntimeError Traceback (most recent call last)
Cell In[3], [line 2](vscode-notebook-cell:?execution_count=3&line=2)
[1](vscode-notebook-cell:?execution_count=3&line=1) from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig, PytorchEngineConfig
----> [2](vscode-notebook-cell:?execution_count=3&line=2) pipe = pipeline(MODEL_PATH,
[3](vscode-notebook-cell:?execution_count=3&line=3) # backend_config=TurbomindEngineConfig(
[4](vscode-notebook-cell:?execution_count=3&line=4) backend_config=PytorchEngineConfig(
[5](vscode-notebook-cell:?execution_count=3&line=5) max_batch_size=32,
[6](vscode-notebook-cell:?execution_count=3&line=6) enable_prefix_caching=True,
[7](vscode-notebook-cell:?execution_count=3&line=7) cache_max_entry_count=0.[8](vscode-notebook-cell:?execution_count=3&line=8),
8 session_len=81[9](vscode-notebook-cell:?execution_count=3&line=9)2,
9 quant_policy=4,
[10](vscode-notebook-cell:?execution_count=3&line=10) ))

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:83, in pipeline(model_path, backend_config, chat_template_config, log_level, max_log_len, **kwargs)
[80](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:80) backend = 'pytorch' if type(backend_config) is PytorchEngineConfig else 'turbomind'
[81](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:81) logger.info(f'Using {backend} engine')
---> [83](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:83) return pipeline_class(model_path,
[84](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:84) backend=backend,
[85](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:85) backend_config=backend_config,
[86](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:86) chat_template_config=chat_template_config,
[87](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:87) max_log_len=max_log_len,
[88](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:88) **kwargs)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:276, in AsyncEngine.__init__(self, model_path, model_name, backend, backend_config, chat_template_config, max_log_len, **kwargs)
[274](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:274) self._build_turbomind(model_path=model_path, backend_config=backend_config, **kwargs)
[275](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:275) elif backend == 'pytorch':
--> [276](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:276) self._build_pytorch(model_path=model_path, backend_config=backend_config, **kwargs)
[277](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:277) else:
[278](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:278) raise ValueError(f'unsupported backend {backend}')

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:328, in AsyncEngine._build_pytorch(self, model_path, backend_config, **kwargs)
[326](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:326) """Innter build method for pytorch backend."""
[327](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:327) from lmdeploy.pytorch.engine import Engine
--> [328](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:328) self.engine = Engine(model_path=model_path, tokenizer=self.tokenizer, engine_config=backend_config)
[329](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:329) self.backend_config = self.engine.engine_config
[330](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:330) self.hf_tm_cfg = getattr(self.engine.model_config, 'hf_config', None)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:148, in Engine.__init__(self, model_path, tokenizer, engine_config, trust_remote_code)
[145](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:145) backend_config = _build_backend_config(engine_config)
[147](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:147) with get_device_manager().context(self.device_context):
--> [148](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:148) self.model_agent = build_model_agent(model_path,
[149](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:149) cache_config=cache_config,
[150](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:150) backend_config=backend_config,
[151](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:151) trust_remote_code=trust_remote_code,
[152](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:152) adapters=adapters,
[153](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:153) tp=self.tp,
[154](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:154) dtype=engine_config.dtype,
[155](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:155) custom_module_map=engine_config.custom_module_map)
[157](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:157) self.input_processor = self.model_agent.get_input_processor()
[159](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:159) cache_config = self.model_agent.cache_config

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:703, in build_model_agent(model_path, cache_config, backend_config, trust_remote_code, adapters, tp, dtype, custom_module_map)
[701](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:701) model_config.custom_module_map = custom_module_map
[702](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:702) if tp == 1:
--> [703](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:703) model_agent = BaseModelAgent(model_path,
[704](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:704) model_config=model_config,
[705](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:705) cache_config=cache_config,
[706](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:706) backend_config=backend_config,
[707](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:707) adapters=adapters,
[708](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:708) trust_remote_code=trust_remote_code)
[709](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:709) else:
[710](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:710) model_agent = TPModelAgent(model_path,
[711](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:711) model_config=model_config,
[712](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:712) cache_config=cache_config,
(...)
[715](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:715) adapters=adapters,
[716](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:716) trust_remote_code=trust_remote_code)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:208, in BaseModelAgent.__init__(self, model_path, model_config, cache_config, backend_config, adapters, trust_remote_code)
[205](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:205) self.backend_config = backend_config
[206](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:206) self._adapters = adapters
--> [208](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:208) self.patched_model = self._build_model(model_path, adapters, device=device)
[210](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:210) _update_cache_config(model_config, cache_config)
[212](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:212) backend = get_backend()

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:229, in BaseModelAgent._build_model(self, model_path, adapters, device)
[227](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:227) update_custom_module_map(custom_module_map)
[228](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:228) logger.info('build model.')
--> [229](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:229) patched_model = build_patched_model(self.model_config, device=device)
[230](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:230) logger.info('loading weights.')
[231](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:231) load_model_weights(patched_model, model_path, device=device)

File ~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:116, in context_decorator..decorate_context(*args, **kwargs)
[113](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:113) @functools.wraps(func)
[114](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:114) def decorate_context(*args, **kwargs):
[115](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:115) with ctx_factory():
--> [116](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:116) return func(*args, **kwargs)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:195, in build_patched_model(config, device)
[193](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:193) model_config = config.hf_config
[194](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:194) dtype = config.dtype
--> [195](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:195) return build_model_from_hf_config(model_config, dtype=dtype, device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:186, in build_model_from_hf_config(model_config, dtype, device)
[184](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:184) device = torch.device('cuda')
[185](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:185) model_cls = _get_model_class(model_config, module_map)
--> [186](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:186) model = model_cls(model_config, ctx_mgr, dtype=dtype, device=device)
[187](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:187) return model.eval()

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:306, in Qwen2ForCausalLM.__init__(self, config, ctx_mgr, dtype, device)
[304](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:304) self.ctx_mgr = ctx_mgr
[305](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:305) # build model
--> [306](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:306) self.model = Qwen2Model(config, dtype=dtype, device=device)
[307](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:307) # build lm_head
[308](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:308) self.lm_head = build_rowwise_linear(config.hidden_size,
[309](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:309) config.vocab_size,
[310](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:310) bias=False,
[311](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:311) dtype=dtype,
[312](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:312) device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:219, in Qwen2Model.__init__(self, config, dtype, device)
[211](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:211) self.embed_tokens = nn.Embedding(config.vocab_size,
[212](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:212) config.hidden_size,
[213](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:213) self.padding_idx,
[214](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:214) dtype=dtype,
[215](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:215) device=device)
[217](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:217) # build all decode layers
[218](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:218) self.layers = nn.ModuleList([
--> [219](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:219) Qwen2DecoderLayer(config, layer_idx, dtype=dtype, device=device)
[220](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:220) for layer_idx in range(config.num_hidden_layers)
[221](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:221) ])
[223](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:223) # build norm
[224](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:224) self.norm = RMSNorm(config.hidden_size, config.rms_norm_eps, dtype=dtype, device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:153, in Qwen2DecoderLayer.__init__(self, config, layer_idx, dtype, device)
[150](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:150) quantization_config = getattr(config, 'quantization_config', None)
[152](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:152) # build attention layer
--> [153](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:153) self.self_attn = Qwen2Attention(config, dtype=dtype, device=device)
[155](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:155) # build MLP
[156](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:156) self.mlp = Qwen2MLP(config, dtype=dtype, device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:30, in Qwen2Attention.__init__(self, config, dtype, device)
[28](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:28) num_replicate_kv_heads = getattr(config, 'num_replicate_key_value_heads', 1)
[29](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:29) # packed qkv
---> [30](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:30) self.qkv_proj = build_qkv_proj(hidden_size,
[31](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:31) num_q_heads=num_heads,
[32](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:32) num_kv_heads=num_key_value_heads,
[33](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:33) head_size=head_dim,
[34](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:34) bias=True,
[35](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:35) quant_config=quantization_config,
[36](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:36) dtype=dtype,
[37](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:37) device=device,
[38](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:38) num_replicate_kv_heads=num_replicate_kv_heads)
[40](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:40) # rotary embedding
[41](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:41) self.apply_rotary_pos_emb = ApplyRotaryEmb()

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1495, in build_qkv_proj(in_features, num_q_heads, num_kv_heads, head_size, head_size_v, bias, quant_config, dtype, device, is_tp, num_replicate_kv_heads)
[1483](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1483) return QKVW8A8Linear(in_features=in_features,
[1484](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1484) num_q_heads=num_q_heads,
[1485](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1485) num_kv_heads=num_kv_heads,
(...)
[1492](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1492) num_replicate_kv_heads=num_replicate_kv_heads,
[1493](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1493) quant_dtype=quant_dtype)
[1494](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1494) else:
-> [1495](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1495) raise RuntimeError(f'Unsupported quant method: {quant_method}')

RuntimeError: Unsupported quant method: gptq
```
换成awq 和 pytorch 以后得到下列错误提示
```
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
```
awq 和 Turbomind同样jupyter崩溃