microsoft / microsoft/MInference

[Question]: ModuleNotFoundError: No module named 'minference.cuda'

Open
#45 2 comments 0 reactions 1 assignee View on GitHub

@iofu728 is already working on this.

Since Jul 17, 2024.

question
Dominant language
Python
Stars
1.2k
Forks
82
Avg merge
1d 18h
Merged PRs (30d)
1

Description

Describe the issue

I encountered some issues when using minference in Python.
import minference
The problem is Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/workspace/MInference/minference/__init__.py", line 8, in <module> from .models_patch import MInference File "/workspace/MInference/minference/models_patch.py", line 7, in <module> from .patch import minference_patch, minference_patch_vllm, patch_hf File "/workspace/MInference/minference/patch.py", line 12, in <module> from .modules.minference_forward import ( File "/workspace/MInference/minference/modules/minference_forward.py", line 20, in <module> from ..ops.pit_sparse_flash_attention_v2 import vertical_slash_sparse_attention File "/workspace/MInference/minference/ops/pit_sparse_flash_attention_v2.py", line 10, in <module> from ..cuda import convert_vertical_slash_indexes ModuleNotFoundError: No module named 'minference.cuda'

enviornment :
Python 3.10.14
minference 0.1.4.post3
triton 2.1.0
torch 2.3.0
CUDA 11.8
vllm 0.4.2+cu118
flash-attn 2.5.8

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.