microsoft / microsoft/fara

vllm serve fails with ModuleNotFoundError: No module named 'vllm._C'

Open
#28 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6.2k
Forks
603
PR merge metrics
No merged PRs in 30d

Description

I am following installation guide on Windows 11 x64.

vllm serve fails

(.venv) PS D:\Experiments\fara> vllm serve "microsoft/Fara-7B" --port 5000 --dtype auto
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\Experiments\fara\.venv\Scripts\vllm.exe\__main__.py", line 2, in <module>
    from vllm.entrypoints.cli.main import main
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\entrypoints\cli\__init__.py", line 3, in <module>
    from vllm.entrypoints.cli.benchmark.latency import BenchmarkLatencySubcommand
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\entrypoints\cli\benchmark\latency.py", line 5, in <module>
    from vllm.benchmarks.latency import add_cli_args, main
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\benchmarks\latency.py", line 17, in <module>
    from vllm.engine.arg_utils import EngineArgs
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\engine\arg_utils.py", line 35, in <module>
    from vllm.attention.backends.registry import AttentionBackendEnum
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\attention\__init__.py", line 4, in <module>
    from vllm.attention.backends.abstract import (
    ...<3 lines>...
    )
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\attention\backends\abstract.py", line 9, in <module>
    from vllm.model_executor.layers.linear import ColumnParallelLinear
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\model_executor\__init__.py", line 4, in <module>
    from vllm.model_executor.parameter import BasevLLMParameter, PackedvLLMParameter
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\model_executor\parameter.py", line 11, in <module>
    from vllm.distributed import (
    ...<2 lines>...
    )
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\distributed\__init__.py", line 4, in <module>
    from .communication_op import *
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\distributed\communication_op.py", line 9, in <module>
    from .parallel_state import get_tp_group
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\distributed\parallel_state.py", line 250, in <module>
    direct_register_custom_op(
    ~~~~~~~~~~~~~~~~~~~~~~~~~^
        op_name="all_reduce",
        ^^^^^^^^^^^^^^^^^^^^^
        op_func=all_reduce,
        ^^^^^^^^^^^^^^^^^^^
        fake_impl=all_reduce_fake,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\utils\torch_utils.py", line 640, in direct_register_custom_op
    from vllm.platforms import current_platform
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\platforms\__init__.py", line 257, in __getattr__
    _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
                        ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\utils\import_utils.py", line 89, in resolve_obj_by_qualname
    module = importlib.import_module(module_name)
  File "C:\Users\rail\AppData\Local\Programs\Python\Python313\Lib\importlib\__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Experiments\fara\.venv\Lib\site-packages\vllm\platforms\cuda.py", line 16, in <module>
    import vllm._C  # noqa
    ^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'vllm._C'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the installation guide and the reported vllm serve command on Windows 11 x64. Trace the import path through the listed vllm.platforms.cuda and vllm._C entry points, then verify the documented platform and installation expectations. Done means the guide or issue outcome clearly explains this invocation and its supported result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.