microsoft / microsoft/LLMLingua
[Question]: Fail to reproduce llmlingua on meetingbank
@pzs19 is already working on this.
Since Jul 30, 2024.
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 428
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 1
Description
Describe the issue
Thanks for the interesting work. I tried to reproduce the results of llmlingua on the meetingbank QA dataset with Mistral-7B as the target LLM.
The small LLM I use is https://huggingface.co/NousResearch/Llama-2-7b-hf
However, the results seem much lower than the reported results in Table 4 of llmlingua2 (around 20 than 50.45 in the paper). Here is my implementation:
compressor = PromptCompressor(
model_name=args.model_name,
model_config={},
use_llmlingua2=False
)
iterative_size = 200
comp_dict = compressor.compress_prompt(
context=origin,
instruction="",
question="",
rate=args.compression_rate,
iterative_size=iterative_size,
context_budget="*2.0",
)
I'm wondering if there is any issue with my implementation?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.