google-deepmind / google-deepmind/deepmind-research

Implementation Typo at NFNet?

Open
#178 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
15.2k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

Hi, should this line (https://github.com/deepmind/deepmind-research/blob/master/nfnets/nf_resnet.py#L181) be
```
out = self.activation(x / self.beta)
```
instead of
```
out = self.activation(x) * self.beta
```
Following the paper, h_{i+1} = h_{i} + \alpha * f_{i} ( h_{i} / beta ), thus the x should be first divided by beta instead with activation? Although these two implementations are the same when `beta=1` and `activation=relu`, they are different in other cases.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.