clab / clab/dynet

why colwise add query vector to key vectors instead of concat in attention example

Open
#1,607 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

https://github.com/clab/dynet/blob/557343455d9cc22227960c219d06611e32cca8d7/examples/sequence-to-sequence/attention.py#L76

[Bahdanau et al., 2015](https://arxiv.org/pdf/1409.0473.pdf) seem to concatenate each key vector with query, but not sum.

Contributor guide

No contributing guide indexed for this repository

Research direction

Open examples/sequence-to-sequence/attention.py at line 76 and compare the operation there with the Bahdanau et al. 2015 formulation linked in the issue. Trace how the key and query vectors are prepared, then determine whether the addition is intentional or inconsistent; done means providing a clear explanation or documenting the confirmed discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.