microsoft/MInference

View on GitHub

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Stars: 1.2k
Forks: 82
Open beginner issues: 0
Indexed issues: 86
Avg merge: 1d 18h
Merged PRs (30d): 1

Dominant language: Python
License: MIT
Last GitHub push: Sep 10, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

86 open issues indexed

[Question]:Code related question: Is the search just for the first batch of dataset? Open

question

microsoft/MInference#91 · 2 comments · 1 assignee · Opened Dec 9, 2024
[Question]: vllm-tp generate can't stop Open

question

microsoft/MInference#90 · 2 comments · 1 assignee · Opened Dec 4, 2024
[Question]: RuntimeError encountered when trying to reproduce results in needle in a haystack Open

question

microsoft/MInference#88 · 3 comments · 1 assignee · Opened Nov 26, 2024
[Question]: How can I reproduce the FullAttention results on the Ruler dataset Open

question

microsoft/MInference#87 · 3 comments · 1 assignee · Opened Nov 25, 2024
[Feature Request]: Is it possible to get the returned logsumexp in streamingllm forward? Open

feature request

microsoft/MInference#85 · 1 assignee · Opened Nov 17, 2024
[Question]: Discrepancy in Pre-filling Time and Memory Consumption on Single A100 Open

question

microsoft/MInference#84 · 3 comments · 1 assignee · Opened Nov 15, 2024
[Question]: Am I using minference correctly? Open

question

microsoft/MInference#83 · 2 comments · 1 assignee · Opened Oct 30, 2024
[Question]: analysis of attention scores (too sparse) Open

question

microsoft/MInference#82 · 2 comments · 1 assignee · Opened Oct 19, 2024
[Question]: sparsity of minference Open

question

microsoft/MInference#78 · 2 comments · 1 assignee · Opened Sep 23, 2024
[Bug]: Torch not found: can't install with pip install (Python 3.12, CUDA 12.6 Update 1, PyTorch 2.4.1) Open

bug

microsoft/MInference#77 · 3 comments · 1 assignee · Opened Sep 20, 2024
[Question]: Could you provide more examples about other attention usage, e.g., dilated1, streaming, snapkv Open

question

microsoft/MInference#76 · 2 comments · 1 assignee · Opened Sep 18, 2024
[Bug]: loc("Minference/minference/ops/pit_sparse_flash_attention_v2.py":110:23): error: operation scheduled before its operands Open

bug

microsoft/MInference#75 · 2 comments · 2 assignees · Opened Sep 18, 2024
[Feature Request]: Support LLaVA Model feature request / Low generation speed Open

feature request

microsoft/MInference#74 · 1 comment · 1 assignee · Opened Sep 18, 2024
[Question]: what is the speedup of attention kernel of current implemetation? Open

question

microsoft/MInference#73 · 1 comment · 1 assignee · Opened Sep 10, 2024
Performance Degradation when Using MInference with Qwen2-7B-Instruct Model Open

question

microsoft/MInference#71 · 1 comment · 1 assignee · Opened Aug 26, 2024
[Bug]: vllm executor.driver_worker. 'RayWorkerWrapper' object has no attribute 'model_runner' Open

bug

microsoft/MInference#67 · 10 comments · 1 assignee · Opened Aug 8, 2024
[Question]: Confusion about Optimal Search Pattern Configuration Open

question

microsoft/MInference#64 · 2 comments · 1 reaction · 1 assignee · Opened Aug 6, 2024
[Question]: It seems that minference does not currently support tensor parallelism under vllm, right? Because in a multi-card environment, the head_id here is incorrect compared to a single card Open

feature request question

microsoft/MInference#62 · 2 comments · 1 assignee · Opened Aug 4, 2024
[Question]: Why is every head config saved with "vertical_and_slash"? Open

question

microsoft/MInference#57 · 3 comments · 1 assignee · Opened Jul 29, 2024
Does MInference supports CUDA11.8? Open

question

microsoft/MInference#56 · 4 comments · 1 assignee · Opened Jul 29, 2024
Shape of slash mismatch when input batchsize > 1 Open

bug

microsoft/MInference#53 · 1 comment · 1 assignee · Opened Jul 23, 2024
[Question]: attn_type="minference" and attn_type= "hf" got different result Open

question

microsoft/MInference#52 · 2 comments · 1 assignee · Opened Jul 21, 2024
[Question]: Question about the settings of vertical_size and slash_size in vertical_and_slash pattern Open

question

microsoft/MInference#47 · 4 comments · 1 assignee · Opened Jul 17, 2024
[Question]: Does vertical_slash_sparse_attention supported to concatenate all batches into a single row for operation like flash_attn_2_cuda.varlen_fwd? Open

question

microsoft/MInference#46 · 1 comment · 1 assignee · Opened Jul 17, 2024
[Question]: ModuleNotFoundError: No module named 'minference.cuda' Open

question

microsoft/MInference#45 · 2 comments · 1 assignee · Opened Jul 16, 2024
[Question]: Why is running MInference/examples/run_vllm.py not as fast as running vllm alone? Open

question

microsoft/MInference#43 · 1 comment · 1 assignee · Opened Jul 16, 2024
[Question]: How does VLLM use MInference through OpenAI Compatible Server? Open

question

microsoft/MInference#40 · 2 comments · 1 assignee · Opened Jul 15, 2024
[Feature Request]: Support Mistral Model Open

feature request

microsoft/MInference#39 · 13 comments · 1 assignee · Opened Jul 15, 2024
[Question]: How can I generate topk_dims_file_path with self-trained model when in searching pattern? Open

question

microsoft/MInference#36 · 5 comments · 1 assignee · Opened Jul 14, 2024
[Question]: The speed of minference in short context Open

question

microsoft/MInference#33 · 1 comment · 1 assignee · Opened Jul 13, 2024
[Feature Request]: Could you please support microsoft/Phi-3-medium-128k-instruct? Thank you! Open

feature request

microsoft/MInference#26 · 1 comment · 1 assignee · Opened Jul 10, 2024
[Question]: vertical slash pattern Open

question

microsoft/MInference#21 · 4 comments · 1 assignee · Opened Jul 6, 2024
[Question]: Question about KV-cache storage Open

feature request question

microsoft/MInference#20 · 5 comments · 1 assignee · Opened Jul 6, 2024
[Question]: MInference Pre filling is slower than the vllm original version Open

question

microsoft/MInference#18 · 6 comments · 1 assignee · Opened Jul 5, 2024
[Question]: Some questions on the code Open

question

microsoft/MInference#17 · 4 comments · 1 assignee · Opened Jul 5, 2024
[Question]: For the tests such as RULER and InfiniteBench mentioned in the paper, what datasets are used to search for patterns? Open

question

microsoft/MInference#16 · 7 comments · 1 assignee · Opened Jul 5, 2024

microsoft/MInference

Get new issues in your inbox