microsoft / microsoft/MInference
[Bug]: vllm executor.driver_worker. 'RayWorkerWrapper' object has no attribute 'model_runner'
@iofu728 is already working on this.
Since Aug 12, 2024.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 82
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 1
Description
Describe the bug
Describe the issue
I encountered some issues when using minference in Python.
[rank0]: Traceback (most recent call last):
[rank0]: File "/data/code/minference/test_minference.py", line 31, in
[rank0]: llm = minference_patch(llm)
[rank0]: File "/data/miniconda3/envs/moe/lib/python3.10/site-packages/minference/models_patch.py", line 39, in call
[rank0]: return self.patch_model(model)
[rank0]: File "/data/miniconda3/envs/moe/lib/python3.10/site-packages/minference/models_patch.py", line 102, in patch_model
[rank0]: model = minference_patch_vllm(model, self.config.config_path)
[rank0]: File "/data/miniconda3/envs/moe/lib/python3.10/site-packages/minference/patch.py", line 1111, in minference_patch_vllm
[rank0]: llm.llm_engine.model_executor.driver_worker.model_runner.model.apply(update_module)
[rank0]: AttributeError: 'RayWorkerWrapper' object has no attribute 'model_runner'
enviornment :
Python 3.10
minference 0.1.5
triton 2.1.0
torch 2.3.0
CUDA 12.1
vllm 0.4.3
flash-attn 2.5.8
Steps to reproduce
No response
Expected Behavior
No response
Logs
No response
Additional Information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.