microsoft / microsoft/MInference

[Question]: How can I reproduce the FullAttention results on the Ruler dataset

Open
#87 3 comments 0 reactions 1 assignee View on GitHub

@iofu728 is already working on this.

Since Nov 26, 2024.

question
Dominant language
Python
Stars
1.2k
Forks
82
Avg merge
1d 18h
Merged PRs (30d)
1

Description

Describe the issue

Thank you for such a solid piece of work. I followed the instructions in the README and ran the following command:
bash run.sh gradientai/Llama-3-8B-Instruct-262k hf ./result

However, I couldn't reproduce the 4K context length results mentioned in the paper.
The results are as shown in the attached image.
Am I doing something wrong?
Image

 0              1              2              3                4                5                6                7                8      9     10     11    12    13

0 Tasks niah_single_1 niah_single_2 niah_single_3 niah_multikey_1 niah_multikey_2 niah_multikey_3 niah_multivalue niah_multiquery vt cwe fwe qa_1 qa_2
1 Score 100.0 100.0 100.0 100.0 100.0 100.0 83.0 100.0 100.0 94.4 93.33 64.0 44.0
2 Nulls 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25 0/25

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.