microsoft / microsoft/LLMLingua

[Bug]: structured_compress_prompt not working correctly with LLMLingua2

Open
#114 5 comments 0 reactions 1 assignee View on GitHub

@SiyunZhao is already working on this.

Since Mar 21, 2024.

feature request
Dominant language
Python
Stars
6.7k
Forks
428
Avg merge
2d 4h
Merged PRs (30d)
1

Description

Describe the bug

Model:

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True, # Whether to use llmlingua-2
)

Prompt:

prompt = """<llmlingua, compress=False> Read the context and answer the question that follows. Don\'t give information outside the document or repeat your findings. Context: </llmlingua><llmlingua, rate=0.5> Context: October 2004\nAs E. B. White said, "good writing is rewriting."  I didn\'t\nrealize this when I was in school.  In writing, as in math and \nscience, they only show you the finished product.\nYou don\'t see all the false starts.  This gives students a\nmisleading view of how things get made.Part of the reason it happens is that writers
don\'t want   \npeople to see their mistakes.  But I\'m willing to let people\nsee an early draft if it will show how much you have\nto rewrite to beat an essay into shape.Below is the oldest version I can find of\nThe Age of the Essay   \n(probably the second or third day), with\ntext that ultimately survived in \nred and text that later\ngot deleted in gray.\nThere seem to be several categories of cuts: things I got wrong,\nthings that seem like bragging, flames,\ndigressions, stretches of awkward prose, and unnecessary words.I discarded more from the beginning.  That\'s\nnot surprising; it takes a while to hit your stride.
</llmlingua><llmlingua, compress=False> Question: What is the best thing to do in San Francisco? Answer: </llmlingua>"""

Code:

compressed_prompt = llm_lingua.structured_compress_prompt(prompt)

Output:

{'compressed_prompt': 'Read context answer question follows. Don\'t give information outside document or repeat findings. Context: October 2004 E. B. White said "good writing is rewriting." didn\'t realize in school. In writing as in math and science only show finished product don\'t see false starts. gives students misleading view of how things made.Part writers don\'t want people to see mistakes willing to let people see early draft if show much to rewrite to beat essay into shape.Below oldest version of The Age of the Essay (probably second or third day), with text survived in red text later deleted in gray. several categories of cuts: things wrong, things bragging, flames digressions awkward prose unnecessary words.I discarded more from beginning. not surprising; takes a while to hit stride. Question: best thing to do in San Francisco? Answer:', 'compressed_prompt_list': ['Read context answer question follows. Don\'t give information outside document or repeat findings. Context: October 2004 E. B. White said "good writing is rewriting." didn\'t realize in school. In writing as in math and science only show finished product don\'t see false starts. gives students misleading view of how things made.Part writers don\'t want people to see mistakes willing to let people see early draft if show much to rewrite to beat essay into shape.Below oldest version of The Age of the Essay (probably second or third day), with text survived in red text later deleted in gray. several categories of cuts: things wrong, things bragging, flames digressions awkward prose unnecessary words.I discarded more from beginning. not surprising; takes a while to hit stride. Question: best thing to do in San Francisco? Answer:'], 'origin_tokens': 250, 'compressed_tokens': 167, 'ratio': '1.5x', 'rate': '66.8%', 'saving': ', Saving $0.0 in GPT-4.'}
Steps to reproduce

Mentioned above.

Expected Behavior

The part which is wrapped in compress=False tag shouldn't have been compressed.

Logs

No response

Additional Information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.