facebookresearch / facebookresearch/SpinQuant
Question about the optimized rotation matrix for Llama3-70B
- Dominant language
- Python
- Stars
- 428
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I tried to reproduce the results of the paper, and got similar results for Llama2-7B, 13B, 70B, and Llama-3 8B.
However, when I tested Llama3-70B using the optimized rotation matrix you provided [[link](https://drive.google.com/drive/folders/1R2zix4qeXBjcmgnJN1rny93cguJ4rEE8)], the result of RTN was as follows:
| Wikitext-2 PPL | paper-reported | Mine | diff. |
| :--- | ---: | ---: | ---: |
| Llama3-70B | 4.1 | **7.5821** | **3.4821** |
I also found out that GPTQ results of Llama3-70B differ from what you reported. (I used W4A4KV4 rotation matrix for RTN, and W16A4KV4 rotation matrix for GPTQ.)
I guess the provided rotation matrices for Llama3-70B is somehow wrong. Could you check this issue, and provide the right rotation matrix for Llama3-70B if possible?
Thank you.
Contributor guide
Assessment
This issue has not been assessed yet.