f-dangel / f-dangel/backpack

Feature request: `BatchL2Grad` for `LayerNorm`

Open
#327 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
617
Forks
57
PR merge metrics
No merged PRs in 30d

Description

Documenting this feature request from @mf-silva as supporting per-sample L2 gradient norms for `LayerNorm` allows estimating importance scores for data points on LLM architectures which often have `LayerNorm`.
A good starting point to implement this is to take a look at the [custom first-order extension example](https://docs.backpack.pt/en/development/use_cases/example_custom_module.html#first-order-extensions) in the docs.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.