facebookresearch / facebookresearch/SpinQuant

Question about the optimized rotation matrix for Llama3-70B

Open
#11 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
428
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Hello,

I tried to reproduce the results of the paper, and got similar results for Llama2-7B, 13B, 70B, and Llama-3 8B.
However, when I tested Llama3-70B using the optimized rotation matrix you provided [[link](https://drive.google.com/drive/folders/1R2zix4qeXBjcmgnJN1rny93cguJ4rEE8)], the result of RTN was as follows:

| Wikitext-2 PPL | paper-reported | Mine | diff. |
| :--- | ---: | ---: | ---: |
| Llama3-70B | 4.1 | **7.5821** | **3.4821** |

I also found out that GPTQ results of Llama3-70B differ from what you reported. (I used W4A4KV4 rotation matrix for RTN, and W16A4KV4 rotation matrix for GPTQ.)
I guess the provided rotation matrices for Llama3-70B is somehow wrong. Could you check this issue, and provide the right rotation matrix for Llama3-70B if possible?

Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.