microsoft / microsoft/LLMLingua
[Bug]: structured_compress_prompt not working correctly with LLMLingua2
Open
@SiyunZhao is already working on this.
Since Mar 21, 2024.
feature request
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 428
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 1
Description
Describe the bug
Model:
from llmlingua import PromptCompressor
llm_lingua = PromptCompressor(
model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
use_llmlingua2=True, # Whether to use llmlingua-2
)
Prompt:
prompt = """<llmlingua, compress=False> Read the context and answer the question that follows. Don\'t give information outside the document or repeat your findings. Context: </llmlingua><llmlingua, rate=0.5> Context: October 2004\nAs E. B. White said, "good writing is rewriting." I didn\'t\nrealize this when I was in school. In writing, as in math and \nscience, they only show you the finished product.\nYou don\'t see all the false starts. This gives students a\nmisleading view of how things get made.Part of the reason it happens is that writers
don\'t want \npeople to see their mistakes. But I\'m willing to let people\nsee an early draft if it will show how much you have\nto rewrite to beat an essay into shape.Below is the oldest version I can find of\nThe Age of the Essay \n(probably the second or third day), with\ntext that ultimately survived in \nred and text that later\ngot deleted in gray.\nThere seem to be several categories of cuts: things I got wrong,\nthings that seem like bragging, flames,\ndigressions, stretches of awkward prose, and unnecessary words.I discarded more from the beginning. That\'s\nnot surprising; it takes a while to hit your stride.
</llmlingua><llmlingua, compress=False> Question: What is the best thing to do in San Francisco? Answer: </llmlingua>"""
Code:
compressed_prompt = llm_lingua.structured_compress_prompt(prompt)
Output:
{'compressed_prompt': 'Read context answer question follows. Don\'t give information outside document or repeat findings. Context: October 2004 E. B. White said "good writing is rewriting." didn\'t realize in school. In writing as in math and science only show finished product don\'t see false starts. gives students misleading view of how things made.Part writers don\'t want people to see mistakes willing to let people see early draft if show much to rewrite to beat essay into shape.Below oldest version of The Age of the Essay (probably second or third day), with text survived in red text later deleted in gray. several categories of cuts: things wrong, things bragging, flames digressions awkward prose unnecessary words.I discarded more from beginning. not surprising; takes a while to hit stride. Question: best thing to do in San Francisco? Answer:', 'compressed_prompt_list': ['Read context answer question follows. Don\'t give information outside document or repeat findings. Context: October 2004 E. B. White said "good writing is rewriting." didn\'t realize in school. In writing as in math and science only show finished product don\'t see false starts. gives students misleading view of how things made.Part writers don\'t want people to see mistakes willing to let people see early draft if show much to rewrite to beat essay into shape.Below oldest version of The Age of the Essay (probably second or third day), with text survived in red text later deleted in gray. several categories of cuts: things wrong, things bragging, flames digressions awkward prose unnecessary words.I discarded more from beginning. not surprising; takes a while to hit stride. Question: best thing to do in San Francisco? Answer:'], 'origin_tokens': 250, 'compressed_tokens': 167, 'ratio': '1.5x', 'rate': '66.8%', 'saving': ', Saving $0.0 in GPT-4.'}
Steps to reproduce
Mentioned above.
Expected Behavior
The part which is wrapped in compress=False tag shouldn't have been compressed.
Logs
No response
Additional Information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.