Support for arbitrary sequence length in FMHA
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
Research direction
Start with run_fmha.py and the linked Apex patch, then compare the existing FMHA implementation with the benchmarked arbitrary-length approach in the gist. Done means supporting arbitrary sequence lengths for head_dim 64 and 128 while preserving the reported efficiency; use the provided benchmark procedure to verify the result.
Written by the indexing model from the issue text.
Description
We experimented with porting existing fmha implementation to efficiently support arbitrary sequence length for head_dim=64/128 and seem to have found a working solution by computing blocks one at a time and accumulating results.
Here's our approach and benchmarks: https://gist.github.com/krunt/72197074816dfe4035fcd9413e4afb22
Here's our implementation (patch to apex): https://github.com/krunt/apex/compare/727a6452c9b781930acee5e24e09efe9360b4890...3655f21606256efdcae092a0b4fd0ac1151d2de8
You can also find the implementation for head dim 128 in the above gist.
Here's how to run/benchmark it: https://github.com/krunt/apex/blob/arbitlen_fmha_headdim_64/run_fmha.py
Authors: @krunt, @TimDettmers, @xtinkt
We also thank @yjk21 @jdemouth and @jaredcasper for helpful discussions
Almost the same idea independently proposed in: https://arxiv.org/abs/2205.14135
Similar idea with full re-materialization proposed in: https://arxiv.org/abs/2112.05682
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/apex
-
Difficulty 4/5 3-5 days Newbie friendliness 64/100
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100