microsoft/MInference

View on GitHub

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Stars: 1.2k
Forks: 82
Open beginner issues: 0
Indexed issues: 86
Avg merge: 1d 18h
Merged PRs (30d): 1

Dominant language: Python
License: MIT
Last GitHub push: Sep 10, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

86 issues indexed so far

[Question]: How do you manage MInference's budget within SCBench? Open

question

Difficulty 5/5 Over a week Newbie friendliness 20/100

microsoft/MInference#212 · Opened Jun 9, 2026
Consider using pre-built Flash Attention kernels via `kernels` Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/MInference#211 · Opened May 28, 2026
Release Qwen2.5-3B and Llama-3.1-8B 512K context checkpoints on Hugging Face Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/MInference#210 · Opened May 20, 2026
[Bug]: AttributeError: 'DynamicCacheWithRepeat' object has no attribute 'key_cache' Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 30/100

microsoft/MInference#207 · Opened Mar 11, 2026
[Question]: Scbench, mulit turn, input=last turn true answers + now turn prompt, but cache is only the 1st turn prompt????? Open

question

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/MInference#202 · Opened Jan 1, 2026
[Question]: Feasibility of evaluating complex KV compression methods via single-turn or batch-input modes Open

question

Difficulty 5/5 Over a week Newbie friendliness 20/100

microsoft/MInference#200 · Opened Dec 12, 2025
[Question]: Out of memory for MInference in e2e benchmark test & NIAH with long context_window Open

question

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/MInference#199 · Opened Dec 2, 2025
[Question]: Broken link in `Leank`: booksum.jsonl.zst dataset URL is inaccessible Open

question

Difficulty 3/5 1-2 days Newbie friendliness 35/100

microsoft/MInference#197 · Opened Nov 25, 2025
[Bug]: _triton_mixed_sparse_attention in pit_sparse_flash_attention_v2.py doesn't work with triton 3.2.0 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/MInference#196 · 1 comment · Opened Nov 20, 2025
[Bug]: Can't run run_infinitebench.py Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/MInference#195 · 1 comment · Opened Nov 3, 2025
[Question]: can TriMix be used in large models, such as deepseek, qwen3-235b? Open

question

Difficulty 4/5 3-5 days Newbie friendliness 20/100

microsoft/MInference#194 · 3 comments · Opened Oct 20, 2025
[Bug]: Unrecognized configuration class when running TriMix example code on Qwen-2.5-VL model Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

microsoft/MInference#191 · Opened Oct 11, 2025
[Question]: No speedup on multi-card with vllm Open

question

Difficulty 5/5 Over a week Newbie friendliness 25/100

microsoft/MInference#188 · Opened Sep 25, 2025
[Feature Request]: Can MInference support chunked prefill? Open

feature request

microsoft/MInference#186 · 5 comments · 1 reaction · 1 assignee · Opened Sep 19, 2025
[Question]: Support for Qwen3 dense models Open

question

Difficulty 5/5 Over a week Newbie friendliness 20/100

microsoft/MInference#184 · 1 reaction · Opened Sep 11, 2025
[Bug]: Implement of TriangleMix Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

microsoft/MInference#183 · 2 comments · Opened Sep 5, 2025
[Bug]: Unable to run streamingLLM properly on minference with the qwen3-0.6B model. Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 30/100

microsoft/MInference#182 · Opened Sep 1, 2025
[Question]: How to use quantification and MInference together Open

question

Difficulty 5/5 Over a week Newbie friendliness 20/100

microsoft/MInference#181 · Opened Aug 27, 2025
[Question]: quick evaluation of kernel Open

question

Difficulty 3/5 1-2 days Newbie friendliness 35/100

microsoft/MInference#180 · Opened Aug 16, 2025
[Bug]: Failed to launch vllm while import MInference Open

bug

microsoft/MInference#178 · 3 comments · 1 assignee · Opened Aug 8, 2025
[Question]: Does Minference support Qwen3 model? Open

question

microsoft/MInference#177 · 2 comments · 1 reaction · 1 assignee · Opened Aug 8, 2025
[Question]: Implement triangle attention in sglang? Open

question

microsoft/MInference#176 · 6 comments · 1 assignee · Opened Aug 7, 2025
[Question]: Is MMinference included? Open

question

microsoft/MInference#175 · 4 comments · 1 assignee · Opened Aug 7, 2025
[Question]: Ruler test results abnormal Open

question

microsoft/MInference#170 · 2 comments · 1 assignee · Opened Jul 22, 2025
[Question]: Setup with VLLM Open

question

microsoft/MInference#169 · 2 comments · 1 assignee · Opened Jul 16, 2025
[Question]: What is the current highest supported version of vLLM? Open

question

microsoft/MInference#168 · 1 comment · 1 assignee · Opened Jul 8, 2025
[Question]: qwen2.5 search pattern config Open

question

microsoft/MInference#167 · 1 comment · 1 reaction · 1 assignee · Opened Jul 4, 2025
[Bug]: OOM occurs when running THUDM/glm-4-9b-chat-1m, both with dense and minference mode. Open

bug

microsoft/MInference#166 · 1 assignee · Opened Jun 24, 2025
[Question]: Top-K Recall Evaluation and Single A100 Inference Details Open

question

microsoft/MInference#164 · 1 comment · 1 assignee · Opened Jun 18, 2025
[ToDo]: V0.1.7 Iteration Plan Open

plan

microsoft/MInference#163 · 4 assignees · Opened Jun 18, 2025
[Question][SCBench]: Questions about disable_golden_context Open

question

microsoft/MInference#154 · 1 comment · 1 assignee · Opened Jun 12, 2025
[Feature Request]: do you support deepseek mla? we need this Open

feature request

microsoft/MInference#151 · 1 assignee · Opened May 19, 2025
[Question]: CUDA out of memory Open

question

microsoft/MInference#150 · 2 comments · 1 assignee · Opened May 19, 2025
[Question]: failed to run the benchmark with error Open

question

microsoft/MInference#145 · 1 comment · 1 assignee · Opened Apr 26, 2025
[Question]: failed to install minference to run the inference benchmark Open

question

microsoft/MInference#143 · 2 comments · 1 assignee · Opened Apr 25, 2025
[Question]: Potential chance for better attention kernel Open

question

microsoft/MInference#140 · 1 comment · 2 reactions · 1 assignee · Opened Apr 24, 2025
[Bug]: Strange Perplexity Results Open

bug

microsoft/MInference#137 · 2 comments · 1 assignee · Opened Apr 16, 2025
[Question]: What is the last stable commit in this repo? Is there any reproducible example? Open

bug question

microsoft/MInference#131 · 1 comment · 1 assignee · Opened Apr 4, 2025
[Bug]: "Illegal memory access was encountered" at vertical_slash_kernel Open

bug

microsoft/MInference#128 · 1 comment · 1 assignee · Opened Mar 14, 2025
[Feature Request]: Do you have any plans to release code for RetrievalAttention? Open

feature request

microsoft/MInference#126 · 2 comments · 1 assignee · Opened Feb 25, 2025
[Question]: e/MInference/scbench/run_scbench.py", line 200, in load_model llm = LLM( NameError: name 'LLM' is not defined. Did you mean: 'llm'? Open

question

microsoft/MInference#123 · 1 comment · 1 assignee · Opened Feb 18, 2025
[Feature Request]: Support for fine-tuning Open

feature request

microsoft/MInference#122 · 1 comment · 1 assignee · Opened Feb 18, 2025
[Question]: Why are the sparse patterns in the provided configuration files all “vertical_and_slash”? Open

question

microsoft/MInference#120 · 4 comments · 1 assignee · Opened Feb 12, 2025
[Bug]: minference package is not installable from sdist Open

bug

microsoft/MInference#118 · 1 assignee · Opened Feb 5, 2025
[Question]: How to find those attention sparse patterns? Open

question

microsoft/MInference#109 · 4 comments · 1 assignee · Opened Jan 7, 2025
[Question]: different eval results compared to the results in paper Open

question

microsoft/MInference#99 · 9 comments · 1 assignee · Opened Dec 19, 2024
[Question]: What are the definitions of the different stages? Open

question

microsoft/MInference#98 · 12 comments · 1 assignee · Opened Dec 19, 2024
[Question]: How to apply MInference on multiple A100 GPUs? Open

question

microsoft/MInference#95 · 1 comment · 1 assignee · Opened Dec 13, 2024
[Question]: How to understand dense_decoding? Open

question

microsoft/MInference#94 · 3 comments · 1 assignee · Opened Dec 13, 2024
[Question]: when searching the best sparse attention type ,why to caculate the score just pick the 2500 cols? Open

question

microsoft/MInference#92 · 2 comments · 1 assignee · Opened Dec 11, 2024

microsoft/MInference

Get new issues in your inbox