kvcache-ai / kvcache-ai/ktransformers

[Feature] Flash Attention Alternative for Older GPU's

Open
#1,385 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [ ] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.

### Motivation

Many older GPU's have enough VRAM but do not support flash attention
EG the Tesla P40

### Related resources

Please consider enabling the Use of vanilla attention as an alternative for those with older GPU's

See this example for another product https://github.com/HiDream-ai/HiDream-I1/pull/27

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Flash Attention integration and the attention entry point in the repository; the issue names no files or tests. Review the linked HiDream-I1 pull request for its vanilla-attention approach, then determine how to verify inference on an older GPU such as the Tesla P40. Done means an alternative attention path is available and works on GPUs without Flash Attention support.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.