google-deepmind / google-deepmind/ai-foundations
Wrong number of parameters for the normalization layers in the transformer block
- Dominant language
- Jupyter Notebook
- Stars
- 305
- Forks
- 153
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/google-deepmind/ai-foundations/blob/e37bc99485767ebe68d8ff4db438721daa7ab966/course_4/gdm_lab_4_5_reflection_on_trainable_parameters.ipynb#L1693
Since there are two layers, shouldn't the number of parameters be:
```
layer_norm_parameter_count = 4 * embedding_dim
```
(layer_1 gamma + beta; layer_2 gamma + beta)
Contributor guide
Research direction
Open course_4/gdm_lab_4_5_reflection_on_trainable_parameters.ipynb at the linked line around 1693 and inspect the transformer block's two normalization layers. Verify the parameter-count calculation against both layers, update the displayed calculation if needed, and rerun the relevant notebook cells to confirm the reported count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100