deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Sparse Attention FP32 support
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
assert query.dtype == torch.half, "sparse attention only supports training in fp16 currently, please file a github issue if you need fp32 support"
Any update on fp32 support for spase attention? Currently I'm happy with training sparse model on fp16 but it seems to me fp32 is more versatile.
I can work on this but I'm not familiar with triton and gpu programming. Could you point out some hints at least?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the sparse-attention code containing the shown dtype assertion and trace the Triton and GPU execution path that currently requires torch.half. Determine the scope of FP32 support and verify that sparse attention training works with FP32 inputs; the payload does not name an implementation file or test to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100