facebookresearch / facebookresearch/SpinQuant
A question regarding the rotation matching pairs
- Dominant language
- Python
- Stars
- 428
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
SpinQuant is a subsequent work to QuaRot. However, we have noticed that the definitions of the rotation matrix pairing details differ between the two papers. In QuaRot, first, there is an online Hadamard operation (with a dimension of head_dim) before o_proj. Secondly, o_weight is fused with a Hadamard matrix ( H ) of the entire tensor dimension, as highlighted in the red box in the figure below.
In SpinQuant, the online Hadamard operation before o_proj is removed. Additionally, o_weight is fused with a Hadamard matrix ( H, now i.e., ) of head_dim dimension.
Why is this the case? Are the rotation matrix pairings in the two papers equivalent?
Contributor guide
Assessment
This issue has not been assessed yet.