NVIDIA / NVIDIA/TransformerEngine
NVFP4 Selective Fwd / Bwd
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 831
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 65
Description
The NVFP4 Pretraining paper observed that in cases where there is a loss gap between NVFP4 and a higher precision baseline that the gap can be closed by switching to higher precision during the last ~10% of training (See Appendix D).
Moreover, they observe that the majority of quantization error results from the forward pass.
What is the recommended way to selectively switch to higher precision only for FProp and while using NVFP4 only for backward?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the NVFP4 Pretraining paper, especially Appendix D, and trace the repository's NVFP4 FProp and backward precision paths. Determine whether the existing API exposes separate precision selection for these paths; done means a documented, supported way to use higher precision for FProp while retaining NVFP4 for backward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100