kvcache-ai / kvcache-ai/ktransformers
[Feature] Flash Attention Alternative for Older GPU's
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [ ] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.
### Motivation
Many older GPU's have enough VRAM but do not support flash attention
EG the Tesla P40
### Related resources
Please consider enabling the Use of vanilla attention as an alternative for those with older GPU's
See this example for another product https://github.com/HiDream-ai/HiDream-I1/pull/27
Contributor guide
Research direction
Start by locating the existing Flash Attention integration and the attention entry point in the repository; the issue names no files or tests. Review the linked HiDream-I1 pull request for its vanilla-attention approach, then determine how to verify inference on an older GPU such as the Tesla P40. Done means an alternative attention path is available and works on GPUs without Flash Attention support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100