huggingface / huggingface/diffusers
Support for Exclusive Self Attention (XSA) in Diffusion Models (DiT, U-Net, etc.)
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
**Is your feature request related to a problem? Please describe.**
Self-attention in diffusion models often produces outputs highly similar to the token’s own value, creating redundancy and under-utilizing context from other patches.
**Describe the solution you’d like.**
Add optional support for **Exclusive Self Attention (XSA)** (arXiv:2603.09078) as a drop-in replacement.
XSA orthogonalizes the attention output to remove the self-component:
```python
# Simplified core idea (per token/head)
z_i = y_i - (y_i @ v_i) / (v_i @ v_i) * v_i
```
Expose via `exclusive_self_attention=True` in `Attention` and transformer blocks.
**Describe alternatives you’ve considered.**
Manual patching or third-party implementations.
**Additional context.**
We’ve Been Doing Attention Wrong (2-Line Fix)
https://youtu.be/2eZKT4H9_iQ
Contributor guide
Research direction
Start by locating the Attention implementation and the transformer blocks used by the diffusion models, then review how attention outputs and value tensors are handled in PyTorch. The work is done when XSA is an optional drop-in mode exposed through exclusive_self_attention=True and its behavior is validated across the relevant attention paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100