[Bug] Qwen3.6-27B-AWQ out of memory with TurboMind engine on V100
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
### Describe the bug
I've download QuantTrio/Qwen3.6-27B-AWQ, and run it with last lmdeloy version(0.12.3), after turbomind convert completed, it's coredump with a cuda oom error, even after I limit max_context_token_num to 2048.
```
ls -lh Qwen3.6-27B-AWQ
rwxrwxrwx 1 xxx xxx 122 Apr 28 10:16 Qwen3.6-27B-AWQ -> /data/xxx/.cache/huggingface/hub/models--QuantTrio--Qwen3.6-27B-AWQ/snapshots/9b507bdc9afafb87b7898700cc2a591aa6639461/
```
run it with lmdeploy 0.12.3
```
lmdeploy --version0.12.3
nvidia-smi | egrep -e "Default| Driver"
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
| N/A 40C P0 36W / 250W | 32004MiB / 32768MiB | 0% Default |
| N/A 35C P0 24W / 250W | 0MiB / 32768MiB | 0% Default |
| N/A 49C P0 39W / 250W | 30418MiB / 32768MiB | 0% Default |
| N/A 55C P0 48W / 250W | 30418MiB / 32768MiB | 0% Default |
```
core dump with out of memory
```
CUDA_VISIBLE_DEVICES=1 lmdeploy serve api_server QuantTrio/Qwen3.6-27B-AWQ
Fetching 25 files: 100%|████████████████████████████████████████| 25/25 [00:00<00:00, 5991.86it/s]
Download complete: : 0.00B [00:00, ?B/s] | 0/25 [00:00
Contributor guide
Assessment
This issue has not been assessed yet.