google-deepmind / google-deepmind/deepmind-research
Question about Average Channel Variance in NFNet
- Dominant language
- Jupyter Notebook
- Stars
- 15.2k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, NFNet is nice work but I still have some questions.
The paper said:
> The variance is reset at each transition block because in these blocks the skip connection is replaced by a convolution operating on a normalized input, undoing any signal growth on the skip path in the preceding blocks.
In resnetV2 Transition Block(as the figure4 in APPENDIX E, the input of shorcut is `BN(ReLU(x))`. I think the input of conv in shortcut connection isn't a `normalized input`, and the relu operation will change the variance of shortcut,
Can you help me out of my confusion？
Contributor guide
Assessment
This issue has not been assessed yet.