microsoft / microsoft/MInference
Consider using pre-built Flash Attention kernels via `kernels`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 82
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 1
Description
Hey,
I am Sayak from the Kernels team at Hugging Face. I noticed that this project uses Flash Attention which includes a long build time. We ship pre-built binaries (which provide bit-exact outputs as the upstream) and thereby, we make it easy to use.
Using FA3 on a supported machine is as easy as:
# make sure `kernels` is installed: `pip install -U kernels`
from kernels import get_kernel
kernel_module = get_kernel("kernels-community/flash-attn3")
flash_attn_func = kernel_module.flash_attn_func
flash_attn_func(...)
Let us know if you'd be interested in this and and we'd be happy to provide a draft of how it would look in your repo.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the existing Flash Attention integration and determine where its build or import path is defined. Compare that path with the kernels package and the kernels-community/flash-attn3 entry point described in the issue. Done means the project can use the pre-built kernel on supported machines without changing its expected outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100