facebookresearch / facebookresearch/SpinQuant

Memory optimization: Avoid expanding group quantization scales to full weight shape ?

Open
#40 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
428
Forks
93
PR merge metrics
No merged PRs in 30d

Description

**Issue:** Group quantization currently expands scales/zeros from compact format [out_features, num_groups, 1] to full weight shape [out_features, in_features], causing massive memory overhead - up to 18GB+ just for scale storage in large models like LLaMA-3 8B.

**Thought:** Keep scales in their natural compact grouped format and update quantization operations to handle reshaping automatically, reducing scale storage memory by 99%+ while maintaining identical quantization behavior. _But will this result any impact while finding the optimized orthogonal rotation matrix?_

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.