[Bug]使用wsl的Ubuntu22.04系统，lmdeploy任何版本使用pytorch引擎部署，加载权重文件非常卡，单个权重文件(144s/it),32B-AWQ启动 一次需要10分钟

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### 目的
使用最新的源码部署QwQ-32B-AWQ与QWEN2.5-VL-7B-AWQ。
原因：发布版0.7.1不支持这两个模型，源码main分支目前已支持，所以采用源码main分支安装在本地环境来部署模型

### Describe the bug

[Bug] 为何我使用最新的代码，**Main分支直接源码安装pip install -e . 后**，使用lmdeploy serve api_server 启动时如此缓慢，启动时长达10分钟才完成

**90%的时间花费在此处：加载模型，且此时CPU占用非常高
Loading weights from safetensors: 0%| | 0/5 [00:00:
2025-03-07 09:00:56,678 - lmdeploy - DEBUG - base.py:39 - Checking :
2025-03-07 09:00:56,678 - lmdeploy - DEBUG - base.py:39 - Checking :
2025-03-07 09:00:57,015 - lmdeploy - DEBUG - triton.py:40 - Checking environment.
2025-03-07 09:00:57,531 - lmdeploy - DEBUG - base.py:39 - Checking :
2025-03-07 09:00:57,532 - lmdeploy - DEBUG - base.py:39 - Checking :
2025-03-07 09:00:57,532 - lmdeploy - DEBUG - base.py:39 - Checking :
2025-03-07 09:00:57,532 - lmdeploy - WARNING - transformers.py:22 - LMDeploy requires transformers version: [4.33.0 ~ 4.46.1], but found version: 4.49.0
2025-03-07 09:00:57,716 - lmdeploy - DEBUG - builder.py:45 - build model config with DefaultModelConfigBuilder
2025-03-07 09:00:57,716 - lmdeploy - DEBUG - config.py:22 - set torch_dtype to float16 for awq.
2025-03-07 09:00:57,717 - lmdeploy - DEBUG - base.py:39 - Checking :
2025-03-07 09:00:57,726 - lmdeploy - DEBUG - builder.py:45 - build model config with DefaultModelConfigBuilder
2025-03-07 09:00:57,726 - lmdeploy - DEBUG - config.py:22 - set torch_dtype to float16 for awq.
2025-03-07 09:00:57,726 - lmdeploy - INFO - __init__.py:81 - Build executor.
2025-03-07 09:00:59,347 - lmdeploy - INFO - base.py:141 - Building Model.
2025-03-07 09:00:59,347 - lmdeploy - DEBUG - model_agent.py:490 - rank[0] - build model.
2025-03-07 09:01:01,170 - lmdeploy - DEBUG - op_backend.py:69 - Op OpType.RotaryEmbedding fallback to default implementation.
2025-03-07 09:01:01,186 - lmdeploy - DEBUG - op_backend.py:69 - Op OpType.Linear fallback to default implementation.
2025-03-07 09:01:01,235 - lmdeploy - DEBUG - model_agent.py:492 - rank[0] - loading weights.
Loading weights from safetensors: 0%| | 0/5 [00:00