google-research / google-research/big_vision
Memory Efficient Attention integration
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 3.5k
- Forks
- 227
- PR merge metrics
- No merged PRs in 30d
Description
Hello, big_vision team!
Thanks for your work on the repository. Looking through the code I noticed that ViT is using classical attention (see line 91 of ViT implementation). It seems like it should be relatively easy to replace current attention implementation with a memory-efficient alternative from flaxformer (line 595 in flaxformer) just passing dot_product_attention_multihead as attention_fn in nn.MultiHeadDotProductAttnetion (line 221 in flax). I think such improvement is worth considering since Flesh Attention authors reported up to x2.4 speedup on long sequences (1k-4k tokens)
What do you think about it? Are there any limitations that make efficient attention integration harder than it seems? I'm not experienced in Jax, so your feedback would be very appreciated
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read big_vision/models/vit.py around line 91, then compare the referenced flaxformer memory_efficient_attention.py implementation with Flax attention.py around line 221. Check compatibility and limitations for ViT attention, and define validation that demonstrates whether the integration improves memory use or speed on long sequences.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100