Removal of multihead attention from activation
Open
enhancement
- Dominant language
- Python
- Stars
- 38
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Per @punkduckable , multihead attention should be implemented as a layer, not an activation function. However, the current implementation simply uses multihead attention as an activation function, which also disrupts overall structure within `MultiLayerPerceptron`.
multihead attention should be removed from activation, and probably implemented as a derived class of latent space.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.