google-deepmind / google-deepmind/sonnet
this portion of attention code looks incorrect
Open
- Dominant language
- Python
- Stars
- 10k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
` attention_mlp = basic.BatchApply(
mlp.MLP([self._mem_size] * self._attention_mlp_layers))`
`for _ in range(self._num_blocks):
attended_memory = self._multihead_attention(memory)`
shouldnt it be this
`attended_memory = memory`
`for _ in range(self._num_blocks):`
`attended_memory = self._multihead_attention(attended_memory)`
i know memory isn't changed in that function too, so isn't this expected to be redundant.
Contributor guide
Assessment
This issue has not been assessed yet.