google / google/flax

The scales field of the NNX WeightNorm module isn't learnable.

Open Beginner friendly
#5,577 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

It seems that `nnx.WeightNorm.scales` field isn't an `nnx.Param` and therefore won't be trainable, unlike what is specified it should be in the original weight normalisation paper: [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://arxiv.org/abs/1602.07868).

### Problem you have encountered:
```python
In [7]: rngs = nnx.Rngs(42)

In [8]: wn = nnx.WeightNorm(nnx.Linear(3, 1, rngs=rngs), use_scale=True, rngs=rngs)

In [9]: wn.scales
Out[9]: {('kernel',): Array([1.], dtype=float32)}

In [10]: nnx.state(wn)
Out[10]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
},
'scales': {
('kernel',): Array([1.], dtype=float32)
}
})

In [11]: nnx.state(wn, nnx.Param)
Out[11]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
}
})
```

### What you expected to happen:
```python
In [15]: wn.scales[('kernel',)] = nnx.Param(wn.scales[('kernel',)])

In [16]: wn.scales
Out[16]:
{('kernel',): Param( # 1 (4 B)
value=Array([1.], dtype=float32)
)}

In [17]: nnx.state(wn)
Out[17]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
},
'scales': {
('kernel',): Param( # 1 (4 B)
value=Array([1.], dtype=float32)
)
}
})

In [18]: nnx.state(wn, nnx.Param)
Out[18]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
},
'scales': {
('kernel',): Param( # 1 (4 B)
value=Array([1.], dtype=float32)
)
}
})
```

Contributor guide

Open the contributing guide

Research direction

Start at the nnx.WeightNorm implementation and inspect how the scales field is initialized and represented. Reproduce the example with nnx.state(wn) and nnx.state(wn, nnx.Param); done means scales is stored as an nnx.Param and appears in the parameter state without requiring manual reassignment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.