why colwise add query vector to key vectors instead of concat in attention example
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/clab/dynet/blob/557343455d9cc22227960c219d06611e32cca8d7/examples/sequence-to-sequence/attention.py#L76
[Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) seem to concatenate each key vector with query, but not sum.
Contributor guide
No contributing guide indexed for this repository
Research direction
Open examples/sequence-to-sequence/attention.py at line 76 and compare the operation there with the Bahdanau et al. 2015 formulation linked in the issue. Trace how the key and query vectors are prepared, then determine whether the addition is intentional or inconsistent; done means providing a clear explanation or documenting the confirmed discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100