baidu / baidu/Dialogue

Why use single head and do not use positional embedding in Attentive module

Open
#7 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
442
Forks
93
PR merge metrics
No merged PRs in 30d

Description

请问一下作者,您这里为什么在Attentive module只是用单头,而且没有使用位置编码。并且用于连接的FFN也设置隐藏层数目也设置成为了和词向量一样的维度。

另外就是还有一个问题在交叉注意力中,
Ui= AttentiveModule(U, R, R),Rl = AttentiveModule(R,U, U)
这个公式的设计大概是一个怎样的想法

刚刚那个问题一不小心手误删掉了,表示尴尬。
麻烦作者解答一下谢谢啦。

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by locating the Attentive module and its FFN, then trace the cross-attention entry points represented by Ui = AttentiveModule(U, R, R) and Rl = AttentiveModule(R, U, U). Done means documenting the rationale for single-head attention, omitting positional embeddings, the FFN dimensions, and the cross-attention design.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.