microsoft / microsoft/LLMLingua

[Question]: LongBench BM25 reproduce

Open
#161 3 comments 0 reactions 1 assignee View on GitHub

@iofu728 is already working on this.

Since Jun 3, 2024.

question
Dominant language
Python
Stars
6.7k
Forks
428
Avg merge
2d 4h
Merged PRs (30d)
1

Description

Describe the issue

I'm interested in your longllmlingua results on LongBench.
I reproduced LongBench BM25 2,000-token constraint using ChatGPT.
Unlike the your paper's results, the performance is too high.
trec task score is 72.5 and most of the other tasks are also high.
I would like to know how you produced the bm25 result.
I'll show you the parameter I used to reproduce bm25 so I'd appreciate it if you could tell me which one is different.
I use same split and parameters other tasks.(only q_format and first inst changing according to original LongBench config)

Thank you

first_inst="Please determine the type of the question below. Here are some examples of questions."
q_format="{input}"
question= q_format.format(input=input)
instruction=first_inst
contexts_list = df['ctxs'][i].split("\n")
contexts_list = [
"\n".join(contexts_list[ii : ii + 4]) for ii in range(0, len(contexts_list), 4)
]
compressed_prompt = llm_lingua.compress_prompt(
contexts_list,
instruction=instruction,
question=question,
target_token=1800,
condition_compare=True,
condition_in_question="after",
rank_method="bm25",
use_sentence_level_filter=False,
use_token_level_filter=False,
context_budget="+100",
dynamic_context_compression_ratio=0.4, # enable dynamic_context_compression_ratio
)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.