Check if fused inplace Relu|LeakyRelu + Dropout works in core and provide recipes / tutorial
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
🚀 The feature
If it works, check it speed and memory usage compared to just inplace relu + outplace dropout
https://github.com/pytorch/pytorch/issues/30801
Manual version is at https://gist.github.com/vadimkantorov/360ece06de4fd2641fa9ed1085f76d48, but it allocates 1 float copy of input + 2 boolean masks. It would be nice to eliminate these allocations by fusion
Motivation, pitch
Given that torchvision started to add things like ConvNormAct, things like fused inplace ReluDropout start to make more sense
Alternatives
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked PyTorch issue #30801 and the manual implementation gist. Check whether fused in-place ReLU/LeakyReLU plus Dropout is supported in core, then compare speed and memory against the unfused variant. Done means the support status and benchmark results are recorded, with recipes or a tutorial provided if applicable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- documentation, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100