microsoft / microsoft/LLMLingua
How to reproduce Multidocument QA results under 9th？
@iofu728 is already working on this.
Since Feb 19, 2024.
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 428
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 1
Description
My reproduction of the results on location 9 of the NQ dataset in the longllmlingua paper using the prompt compressor resulted in a large discrepancy from the original results. My hyperparameters are set as follows：
The args.t was set to True and False in two experiments, which was to verify the validity of the contrast ITC.When args.t is set to True, accuracy is 63, while when args.t is set to False, the accuracy is 69.
Questions:
1.What are the hyperparameters that can accurately reproduce the results in the paper with an accuracy of approximately 70.8%(NQ 2x 9th)?
2.Why does contrast ITC drop so severely under my current settings?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.