Why are query, key, and value all set to the same value?
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.3k
- Forks
- 379
- Avg merge
- 3h 30m
- Merged PRs (30d)
- 8
Description
Multi-head self-attention output (tf.keras.layers.MultiHeadAttention ).
attn_output = self.mha(
query=x, # Query Q tensor.
value=x, # Value V tensor.
key=x, # Key K tensor.
attention_mask=attention_mask, # A boolean mask that prevents attention to certain positions.
training=training, # A boolean indicating whether the layer should behave in training mode.
)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tf.keras.layers.MultiHeadAttention call shown in the issue and consult the layer's documented query, key, and value arguments. Clarify why the self-attention example passes x for all three inputs, and update the relevant explanation or documentation once the intended behavior is confirmed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100