huggingface / huggingface/diffusers

[optimization] help us know which kernels we should integrate in Diffusers

Open
#12,990 13 comments 1 reaction 0 assignees View on GitHub
performance
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

This issue is for knowing which kernels we should integrate into the library through [`kernels`](https://github.com/huggingface/kernels/).

Currently, we leverage `kernels` for different attention backends (FA2, FA3, and SAGE). However, other layers can be optimized as well (RMS Norm, for example), depending on the model size and input payload being used to benchmark that.

I did take a crack at this once, i.e., replacing the norm layers with their optimized counterparts, but didn't realize any noticeable gains. But maybe this is different now.

## Resources / notes

* There's a bunch of kernels we maintain in https://huggingface.co/kernels-community, which could be repurposed in this case.
* We can also work together with the community to port impactful kernels and host them through https://huggingface.co/kernels-community and make sure they are `kernels` compatible.
* `transformers` gain benefits from this paradigm in terms of latency improvements (@MekkCyber can provide details). Some relevant PRs can be found here: https://github.com/huggingface/transformers/commits?author=MekkCyber (look for PRs with titles starting with "[kernels]")

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing the existing kernels integration for FA2, FA3, and SAGE, then benchmark candidate layers such as RMS Norm across model sizes and input payloads. Done means identifying kernels with noticeable gains and determining which are suitable for integration through kernels.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.