f-dangel / f-dangel/backpack

Support for LayerNorm

Open
#263 3 comments 1 reaction 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
617
Forks
57
PR merge metrics
No merged PRs in 30d

Description

I was trying to extend a Vision Transformer model using backpack. However, I encounter the following error:

> UserWarning: Extension saving to grad_batch does not have an extension for Module although the module has parameters

I know that torch.nn.BatchNormNd leads to ill-defined first-order quantities and hence it is not implemented here. Does the same hold for Layer Normalization?

Thank you in advance!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.