google-deepmind / google-deepmind/ai-foundations

Wrong number of parameters for the normalization layers in the transformer block

Open Beginner friendly
#37 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
305
Forks
153
PR merge metrics
No merged PRs in 30d

Description

https://github.com/google-deepmind/ai-foundations/blob/e37bc99485767ebe68d8ff4db438721daa7ab966/course_4/gdm_lab_4_5_reflection_on_trainable_parameters.ipynb#L1693

Since there are two layers, shouldn't the number of parameters be:

```
layer_norm_parameter_count = 4 * embedding_dim
```

(layer_1 gamma + beta; layer_2 gamma + beta)

Contributor guide

Open the contributing guide

Research direction

Open course_4/gdm_lab_4_5_reflection_on_trainable_parameters.ipynb at the linked line around 1693 and inspect the transformer block's two normalization layers. Verify the parameter-count calculation against both layers, update the displayed calculation if needed, and rerun the relevant notebook cells to confirm the reported count.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
machine-learning
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.