Feature request: `BatchL2Grad` for `LayerNorm`
Open
enhancement
good first issue
- Dominant language
- Python
- Stars
- 617
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
Documenting this feature request from @mf-silva as supporting per-sample L2 gradient norms for `LayerNorm` allows estimating importance scores for data points on LLM architectures which often have `LayerNorm`.
A good starting point to implement this is to take a look at the [custom first-order extension example](https://docs.backpack.pt/en/development/use_cases/example_custom_module.html#first-order-extensions) in the docs.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.