microsoft/LLMLingua

View on GitHub

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

Stars: 6.7k
Forks: 428
Open beginner issues: 0
Indexed issues: 101
Avg merge: 2d 4h
Merged PRs (30d): 1

Dominant language: Python
License: MIT
Last GitHub push: Sep 10, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

101 open issues indexed

Code Review Observations & Questions from a Developer Exploring the Project Open

Difficulty 5/5 Over a week Newbie friendliness 42/100

microsoft/LLMLingua#249 · 1 reaction · Opened Jun 10, 2026
[Bug]: When calling slingua through PromptCompressor, use_llmlingua2 should also be set to true. Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

microsoft/LLMLingua#235 · Opened Jan 29, 2026
[Question]: Is this repo still relevant? Open

question

Difficulty 1/5 Under an hour Newbie friendliness 20/100

microsoft/LLMLingua#234 · 2 comments · Opened Jan 20, 2026
[Bug]: XLMRobertaForTokenClassification.forward() got an unexpected keyword argument 'past_key_values' Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

microsoft/LLMLingua#232 · 1 comment · 1 reaction · Opened Dec 25, 2025
[Question]: Significant performance drop on LongBench using LongChat-7b-v1.5-32k compared to official full-context baseline Open

question

Difficulty 4/5 3-5 days Newbie friendliness 25/100

microsoft/LLMLingua#231 · Opened Dec 11, 2025
[Question]: How to obtain the masked and kept tokens? Open

question

Difficulty 3/5 1-2 days Newbie friendliness 35/100

microsoft/LLMLingua#229 · 1 comment · Opened Sep 9, 2025
[Question]: force_tokens for reproducing LLMLingua-2 on LongBench DataSet Open

question

Difficulty 3/5 1-2 days Newbie friendliness 25/100

microsoft/LLMLingua#228 · Opened Sep 1, 2025
[Question]: Is it possible to retrieve the word probability and threshold? Open

question

Difficulty 5/5 Over a week Newbie friendliness 25/100

microsoft/LLMLingua#227 · Opened Aug 19, 2025
[Question]：how to use remote model service，such that alibaba modelscope Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

microsoft/LLMLingua#225 · Opened Jul 14, 2025
[Question]: why ？NotImplementedError Open

question

microsoft/LLMLingua#221 · 1 comment · 1 assignee · Opened Jun 26, 2025
Release SecurityLingua on Hugging Face Open

feature

microsoft/LLMLingua#220 · 2 comments · 1 reaction · 1 assignee · Opened Jun 21, 2025
[Question]:Is there a supported model list? Open

question

microsoft/LLMLingua#218 · 1 comment · 1 assignee · Opened Jun 13, 2025
请问可以使用qwen系列模型吗？或者有提供支持模型的列表吗？ Open

question

microsoft/LLMLingua#217 · 1 comment · 1 assignee · Opened May 12, 2025
[BUG] LLMLingua fails on Windows with PyTorch CPU: "Torch not compiled with CUDA enabled" Open

question

microsoft/LLMLingua#216 · 2 comments · 1 assignee · Opened Apr 17, 2025
[Question]: Error when running the tutorial Open

question

microsoft/LLMLingua#214 · 1 comment · 1 assignee · Opened Mar 29, 2025
[Question]: Prompt Compression Issue – Important Information Missing Open

question

microsoft/LLMLingua#213 · 1 comment · 1 assignee · Opened Mar 27, 2025
[Question]: sort_direct = -1 if condition_in_question == "none" else 1 Open

question

microsoft/LLMLingua#212 · 1 comment · 1 assignee · Opened Mar 20, 2025
[Question]: error when compressing long prompt with LLMLingua1 & LongLLMLingua Open

bug question

microsoft/LLMLingua#210 · 4 comments · 1 reaction · 1 assignee · Opened Mar 6, 2025
[Question]: How was edit sim for code tasks calculated? Open

question

microsoft/LLMLingua#207 · 1 comment · 1 assignee · Opened Jan 9, 2025
[Question]: Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'. Open

question

microsoft/LLMLingua#206 · 1 comment · 1 assignee · Opened Dec 19, 2024
Does llmlingua support Markdown format. Open

question

microsoft/LLMLingua#202 · 1 comment · 1 assignee · Opened Nov 27, 2024
[Bug]: regex pattern does not handle nested tags in prompt (structured compression) Open

bug

microsoft/LLMLingua#201 · 1 comment · 1 assignee · Opened Nov 20, 2024
[Bug]: Prompts smaller than iterative_size are not compressed Open

bug

microsoft/LLMLingua#196 · 1 comment · 1 assignee · Opened Nov 14, 2024
[Bug]: Achieved compression rate with (Long)LLMLingua overshot Open

question

Difficulty 4/5 3-5 days Newbie friendliness 28/100

microsoft/LLMLingua#195 · 2 comments · Opened Nov 14, 2024
[Question]: Reproduce end2end latency results of LLMLingua-2 Open

question

Difficulty 5/5 Over a week Newbie friendliness 15/100

microsoft/LLMLingua#193 · 3 comments · Opened Oct 23, 2024
[Question]: 在生成摘要的任务，是如何进行压缩的？ Open

question

microsoft/LLMLingua#190 · 2 comments · 1 assignee · Opened Oct 9, 2024
what is 1-shot / half-shot /quarter-shot constraint in experiments？ Open

question

microsoft/LLMLingua#185 · 7 comments · 1 assignee · Opened Sep 23, 2024
[Bug]: ZeroDivisionError: float division by zero upon using compress_json with `use_llmlingua2` Open

bug

microsoft/LLMLingua#183 · 1 comment · 1 assignee · Opened Sep 17, 2024
[Bug]: LLMLingua-2 uses wrong special tokens by default Open

bug

microsoft/LLMLingua#181 · 2 comments · 1 assignee · Opened Sep 13, 2024
[Bug]: Promptflow tool fails with Run failed: KeyError: 0 Open

bug

microsoft/LLMLingua#178 · 1 comment · 1 assignee · Opened Sep 9, 2024
[Bug]: Had to make a number of changes to example (examples/RAGLlamaIndex.ipynb) Open

bug

microsoft/LLMLingua#177 · 1 assignee · Opened Aug 26, 2024
How can I speed up using llmlingua2 ? Open

question

microsoft/LLMLingua#175 · 1 comment · 1 assignee · Opened Aug 6, 2024
[Question]: LLMLingua-2 Sample-Wise Dynamic Compression Ratio Open

question

microsoft/LLMLingua#174 · 2 comments · 2 assignees · Opened Aug 2, 2024
[Question]: How to utilize compression to a finetuned LLM? Open

question

microsoft/LLMLingua#172 · 1 comment · 1 assignee · Opened Jul 29, 2024
[Question]: Fail to reproduce llmlingua on meetingbank Open

question

microsoft/LLMLingua#171 · 1 comment · 1 assignee · Opened Jul 27, 2024
[Question]: How to get `meetingbank_test_3qa_pairs_summary_formated.json`? Open

question

microsoft/LLMLingua#170 · 2 comments · 1 assignee · Opened Jul 24, 2024
[Question]: Is It Possible to Share the Code how to Do the evaluation Open

question

microsoft/LLMLingua#169 · 3 comments · 1 assignee · Opened Jul 10, 2024
[Bug]: NotImplementedError Open

bug

microsoft/LLMLingua#168 · 4 comments · 1 assignee · Opened Jul 9, 2024
[Question]: Token indices sequence length is longer than the specified maximum sequence length for this model (614 > 512). Running this sequence through the model will result in indexing errors Open

question

microsoft/LLMLingua#165 · 3 comments · 1 assignee · Opened Jun 19, 2024
[Question]: longbench数据集测试对context和question的压缩 Open

question

microsoft/LLMLingua#163 · 2 comments · 1 assignee · Opened Jun 5, 2024
[Question]: How to combine longllmlingua and llmlingua2 ? Open

question

microsoft/LLMLingua#162 · 1 comment · 1 assignee · Opened Jun 5, 2024
[Question]: LongBench BM25 reproduce Open

question

microsoft/LLMLingua#161 · 3 comments · 1 assignee · Opened May 30, 2024
[Feature Request]: How to improve the accuracy of compressor for large SFT models through training Open

feature request

microsoft/LLMLingua#159 · 1 comment · 1 assignee · Opened May 29, 2024
[Question]: Reproducing the score of official microsoft/llmlingua-2-xlm-roberta-large-meetingbank Open

question

microsoft/LLMLingua#156 · 6 comments · 2 assignees · Opened May 23, 2024
[Question]: Reproduce LLMLingua-2 results with Mistral-7B Open

question

microsoft/LLMLingua#155 · 4 comments · 2 assignees · Opened May 21, 2024
[Bug]: 返回中文会出现乱码 Open

bug

microsoft/LLMLingua#152 · 2 comments · 1 assignee · Opened May 13, 2024
[Feature Request]: Lingua2 can discards tokens based on a probability threshold Open

feature request

microsoft/LLMLingua#150 · 1 comment · 2 assignees · Opened May 9, 2024
Meaningless tokens generation Open

question

microsoft/LLMLingua#149 · 3 comments · 1 assignee · Opened May 9, 2024
[Question]: LLMLingua requires too much GPU memory, and it takes a lot of time to compress long text, such as 16k, etc. How to make it and LLM work at the same time Open

question

microsoft/LLMLingua#147 · 2 comments · 1 assignee · Opened May 9, 2024
[Question]: Reproduce LLMLingua-2 on the LongBench SingleDoc dataset Open

question

microsoft/LLMLingua#146 · 2 comments · 2 assignees · Opened May 7, 2024

microsoft/LLMLingua

Get new issues in your inbox