The scales field of the NNX WeightNorm module isn't learnable.
- Dominant language
- Jupyter Notebook
- Stars
- 7.3k
- Forks
- 833
- Avg merge
- 5h 11m
- Merged PRs (30d)
- 5
Description
It seems that `nnx.WeightNorm.scales` field isn't an `nnx.Param` and therefore won't be trainable, unlike what is specified it should be in the original weight normalisation paper: [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://arxiv.org/abs/1602.07868).
### Problem you have encountered:
```python
In [7]: rngs = nnx.Rngs(42)
In [8]: wn = nnx.WeightNorm(nnx.Linear(3, 1, rngs=rngs), use_scale=True, rngs=rngs)
In [9]: wn.scales
Out[9]: {('kernel',): Array([1.], dtype=float32)}
In [10]: nnx.state(wn)
Out[10]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
},
'scales': {
('kernel',): Array([1.], dtype=float32)
}
})
In [11]: nnx.state(wn, nnx.Param)
Out[11]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
}
})
```
### What you expected to happen:
```python
In [15]: wn.scales[('kernel',)] = nnx.Param(wn.scales[('kernel',)])
In [16]: wn.scales
Out[16]:
{('kernel',): Param( # 1 (4 B)
value=Array([1.], dtype=float32)
)}
In [17]: nnx.state(wn)
Out[17]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
},
'scales': {
('kernel',): Param( # 1 (4 B)
value=Array([1.], dtype=float32)
)
}
})
In [18]: nnx.state(wn, nnx.Param)
Out[18]:
State({
'layer_instance': {
'bias': Param( # 1 (4 B)
value=Array([0.], dtype=float32)
),
'kernel': Param( # 3 (12 B)
value=Array([[ 0.04756278],
[-0.30357933],
[ 0.782016 ]], dtype=float32)
)
},
'scales': {
('kernel',): Param( # 1 (4 B)
value=Array([1.], dtype=float32)
)
}
})
```
Contributor guide
Research direction
Start at the nnx.WeightNorm implementation and inspect how the scales field is initialized and represented. Reproduce the example with nnx.state(wn) and nnx.state(wn, nnx.Param); done means scales is stored as an nnx.Param and appears in the parameter state without requiring manual reassignment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100