Question about Memory-efficient self-attention
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 298
- Forks
- 26
- Avg merge
- 22h 44m
- Merged PRs (30d)
- 1
Description
Hi, it is a nice job about utilizing swin transformer to point cloud. However, I really don't understand the content of Memory-efficient self-attention.
$f_{i,h}^{*}=\frac{\sum_{j=1}^{N}(exp(e_{ij,h})f_{j}W_{V,h})}{\sum_{j=1}^{N}exp(e_{ij},h)}----(3)$
how can I understand the idea of allowing to postpone the SoftMax normalization and avoid constructing and storing ${αij,h}$ explicitly.
Calculating the denominator and numerator of Eq. (3) simultaneously is also a question that hard to fully understand.
Could you please give me some tips about how to grasp the idea of Memory-efficient self-attention.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are mentioned. First clarify whether the desired outcome is documentation explaining the memory-efficient self-attention equations or an implementation change; done should be a specific, reviewed explanation or a clearly scoped code task.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100