NVIDIA / NVIDIA/TransformerEngine
Question regarding the 2D weight quantization and quality degradation in 1D case
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 831
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 65
Description
Many thanks for the great work!
In the paper https://arxiv.org/pdf/2502.20853 they use 1D weight quantization with requantization with success, and also from their repo https://github.com/thu-ml/TetraJet-MXFP4Training/issues/2#issuecomment-3454394125 the author mentioned from their experience 1D weight quantization produce better result than 2D weight quantization.
So I was wondering how did you implement the 1D weight quantization (in the transpose dimension in bwd) in your paper that produces worse result compared to 2D case? Did you also use requantization (i.e. quantize then dequantize then transpose then quantize) in 1D weight quantization?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or entry point is named. Start by reading the referenced paper and the linked TetraJet-MXFP4Training issue comment, then trace TransformerEngine's 1D and 2D weight-quantization paths to determine whether requantization is used; done means documenting the implementation details and explaining the reported quality difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100