shape of dist_matrix in MultiSelfAttention
Open
- Dominant language
- Python
- Stars
- 435
- Forks
- 118
- PR merge metrics
- No merged PRs in 30d
Description
In MultiSelfAttention class:
line 235: dist_matrix = tf.einsum("bwhd,bkhd->bwkh", queries, keys) # dots of (batch, word, key, head)
line 255: select_probs = tf.nn.softmax(dist_matrix) # for each (batch, word, head) probability over keys
The shapes of dist_matrix in the two lines seem not consistent. Should line 235 be changed to:
dist_matrix = tf.einsum("bwhd,bkhd->bwhk", queries, keys) # dots of (batch, word, head, key) ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.