facebookresearch / facebookresearch/SpinQuant
Memory optimization: Avoid expanding group quantization scales to full weight shape ?
Open
- Dominant language
- Python
- Stars
- 428
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
**Issue:** Group quantization currently expands scales/zeros from compact format [out_features, num_groups, 1] to full weight shape [out_features, in_features], causing massive memory overhead - up to 18GB+ just for scale storage in large models like LLaMA-3 8B.
**Thought:** Keep scales in their natural compact grouped format and update quantization operations to handle reshaping automatically, reducing scale storage memory by 99%+ while maintaining identical quantization behavior. _But will this result any impact while finding the optimized orthogonal rotation matrix?_
Contributor guide
Assessment
This issue has not been assessed yet.