[Bug] qwen3-awq internal error happened
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
### Describe the bug
30906
ChatCompletionMessage(content='internal error happened', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None, reasoning_content=None)
## Issues:
1. The model throws an error when the token length is slightly longer, which is puzzling since the current VRAM should be capable of handling contexts of around 60,000 tokens.
2. The concurrency performance of the Qwen3 series models on 20-series and 30-series GPUs is significantly lower compared to Qwen2.5.
### Reproduction
CUDA_VISIBLE_DEVICES=0 lmdeploy serve api_server /home/ma/work/modelscope/Qwen3-8B-AWQ --backend turbomind --model-format awq --server-port 1251 --session-len 60000 --cache-max-entry-count 0.5 --model-name gpt --tp 1
### Environment
```Shell
Driver Version: 555.42.06 CUDA Version: 12.5
Name: torch
Version: 2.5.1
Name: lmdeploy
Version: 0.8.0
```
### Error traceback
```Shell
```
Contributor guide
Assessment
This issue has not been assessed yet.