modelscope / modelscope/ms-swift
The max_num environment variable does not work when using vllm as the backend
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)
在使用InternVL3-8B进行推理时，允许使用export MAX_NUM=6来控制动态高分辨率的子图数量。在使用pt作为backend时该参数能正常起作用（不会超长），但在设置--infer_backend vllm后该参数不起作用，会出现token过长的报错：ValueError: The decoder prompt (length 35191) is longer than the maximum model length of 32768.
Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)
该现象与CUDA无关
Additional context
Add any other context about the problem here(在这里补充其他信息)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named. Reproduce with InternVL3-8B using MAX_NUM=6 first with the pt backend, then with --infer_backend vllm, and trace how the setting reaches the vLLM inference path. Done means the vLLM path respects the subimage limit and avoids the maximum-model-length error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100