Lightning-AI / Lightning-AI/pytorch-lightning
shortcuts for logging weights and biases norms
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Knowing the norm of weights was necessary to debug float16 training for me.
### Pitch
from lightning.pytorch.utilities import grad_norm
norms = grad_norm(self.layer, norm_type=2)
something like this for weights would be convenient.
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing lightning.pytorch.utilities.grad_norm helper and how it is used for gradient diagnostics. Determine the corresponding weight-norm API and its intended inputs from that entry point. Done means users can conveniently obtain weight and bias norms for debugging float16 training, with behavior covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100