NVIDIA / NVIDIA/TransformerEngine
Request for Adaptive Layer Norm MLP
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 831
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 65
Description
It'd be amazing to have support for a pytorch LayerNormMLP implementation that supports a scale and offset tensor to be applied after the layernorm but before the MLP. Would be curious to hear what it would take to implement this! happy to help
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the PyTorch LayerNormMLP implementation and its existing tests. Trace where layer normalization feeds the MLP, then determine how scale and offset tensors should be accepted and applied between those stages. Done means the requested behavior is supported and covered by tests, including the relevant tensor shapes and execution paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100