google / google/flax

Standardizing normalization layers

Open
#3,664 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

![norm](https://github.com/google/flax/assets/19753743/17bebc7b-c78c-4288-b101-258ea6ef7dbf)

`LayerNorm` is understood as normalization the activations by reducing across all non-batch axes. Currently Flax's implementation of `LayerNorm`, the default [`reduction_axes=-1`](https://github.com/google/flax/blob/main/flax/linen/normalization.py#L430). This works for 2D inputs, but for higher dimensional tensors, this would only reduce the trailing dimension. Should we change the default implementation so that it normalizes all non-batch axes by default (assuming the leading dimension is the batch axes)? This also applies to `RMSNorm` as well.

Another thing is that currently all normalization layers with learnable scale and bias have a `feature_axis` (or equivalent) input arg so that the user can specify the shape of the learnable params, except `GroupNorm` (which always defines [`feature_axis=-1`](https://github.com/google/flax/blob/main/flax/linen/normalization.py#L660)). Should we add this into `GroupNorm` as well?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.