Why use single head and do not use positional embedding in Attentive module
Open
- Dominant language
- Python
- Stars
- 442
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
请问一下作者,您这里为什么在Attentive module只是用单头,而且没有使用位置编码。并且用于连接的FFN也设置隐藏层数目也设置成为了和词向量一样的维度。
另外就是还有一个问题在交叉注意力中,
Ui= AttentiveModule(U, R, R),Rl = AttentiveModule(R,U, U)
这个公式的设计大概是一个怎样的想法
刚刚那个问题一不小心手误删掉了,表示尴尬。
麻烦作者解答一下谢谢啦。
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Start by locating the Attentive module and its FFN, then trace the cross-attention entry points represented by Ui = AttentiveModule(U, R, R) and Rl = AttentiveModule(R, U, U). Done means documenting the rationale for single-head attention, omitting positional embeddings, the FFN dimensions, and the cross-attention design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100