[FEA] Add CopyReduceBulkS2G
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Which component requires the feature?
CuTe DSL
Feature Request
Is your feature request related to a problem? Please describe.
There's already CopyBulkS2G and CopyReduceBulkTensorS2G, but there's no CopyReduceBulkS2G (i.e. cp.reduce.async.bulk.global.share ptx instruction).
We use this in Flashattention for dQ reduction. Currently the workaround is to call ptx directly but it would be much better to have a CopyOp for this.
https://github.com/Dao-AILab/flash-attention/blob/c4d8b0630eb81cf88206e0cc9e9bff4e7806d88f/flash_attn/cute/flash_bwd_sm100.py#L2563
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing CopyBulkS2G and CopyReduceBulkTensorS2G implementations in CUTLASS and compare them with the direct PTX use in flash_attn/cute/flash_bwd_sm100.py. Add support for the cp.reduce.async.bulk.global.share instruction through a CopyOp, then verify it against the existing related operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- hpc
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100