microsoft / microsoft/MInference
[Question]: How can I reproduce the FullAttention results on the Ruler dataset
@iofu728 is already working on this.
Since Nov 26, 2024.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 82
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 1
Description
Describe the issue
Thank you for such a solid piece of work. I followed the instructions in the README and ran the following command:
bash run.sh gradientai/Llama-3-8B-Instruct-262k hf ./result
However, I couldn't reproduce the 4K context length results mentioned in the paper.
The results are as shown in the attached image.
Am I doing something wrong?
0 1 2 3 4 5 6 7 8 9 10 11 12 13
0 Tasks niah_single_1 niah_single_2 niah_single_3 niah_multikey_1 niah_multikey_2 niah_multikey_3 niah_multivalue niah_multiquery vt cwe fwe qa_1 qa_2
1 Score 100.0 100.0 100.0 100.0 100.0 100.0 83.0 100.0 100.0 94.4 93.33 64.0 44.0
2 Nulls 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.