Clarification of RMSNorm layer fusion

Hi,

Thanks very much for your work and for publishing your code. I am currently working on integration of SpinQuant into [torch/ao](https://github.com/pytorch/ao/pull/983/), and I would like to clarify something about the code that would help me in my implementation.

In the paper, the following is mentioned in footnote 3:

> In a pre-norm LLM like LLaMA, we can convert a transformer network into a rotation-invariant network by incorporating the RMSNorm scale parameters α into the weight matrix right after the RMSNorm layer.

In the code, this appears to be done in the [fuse_layer_norms](https://github.com/facebookresearch/SpinQuant/blob/7f5bf66a4f5402f3fbd6ffd7ae3f59e175142aed/utils/fuse_norm_utils.py#L39) function.

However, I also noticed that in that same function, the embedding weights are modified, in the following lines:

https://github.com/facebookresearch/SpinQuant/blob/7f5bf66a4f5402f3fbd6ffd7ae3f59e175142aed/utils/fuse_norm_utils.py#L42-L45

Could you help me understand why this is done? I.e. subtraction of the mean from the input embeddings. I don't see a connection to the RMSNorm layer fusion, so I must be missing something.

Thanks in advance.