microsoft / microsoft/TransformerCompression
Convert Layernorm to RMSnorm
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 460
- Forks
- 58
- Avg merge
- 8h 40m
- Merged PRs (30d)
- 1
Description
Hello, I have read your paper "SliceGPT: Compress Large Language Models by Deleting Rows and Columns". In section 3.2, I noticed that you convert LayerNorm into RMSNorm and merged two parts through linear projection. However, I would like to ask how the conversion process ensures that the output remains unchanged without any additional processing of the residual connections.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading section 3.2 of the SliceGPT paper and locating the repository's LayerNorm-to-RMSNorm conversion implementation. Trace how the linear projection and residual connections are handled, then document how output equivalence is maintained or identify what remains unexplained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100