huggingface / huggingface/diffusers

Support for Exclusive Self Attention (XSA) in Diffusion Models (DiT, U-Net, etc.)

Open
#13,447 0 comments 0 reactions 0 assignees View on GitHub
feature-request models
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
Self-attention in diffusion models often produces outputs highly similar to the token’s own value, creating redundancy and under-utilizing context from other patches.

**Describe the solution you’d like.**
Add optional support for **Exclusive Self Attention (XSA)** (arXiv:2603.09078) as a drop-in replacement.

XSA orthogonalizes the attention output to remove the self-component:

```python

# Simplified core idea (per token/head)
z_i = y_i - (y_i @ v_i) / (v_i @ v_i) * v_i

```

Expose via `exclusive_self_attention=True` in `Attention` and transformer blocks.

**Describe alternatives you’ve considered.**
Manual patching or third-party implementations.

**Additional context.**

We’ve Been Doing Attention Wrong (2-Line Fix)

https://youtu.be/2eZKT4H9_iQ

Contributor guide

Open the contributing guide

Research direction

Start by locating the Attention implementation and the transformer blocks used by the diffusion models, then review how attention outputs and value tensors are handled in PyTorch. The work is done when XSA is an optional drop-in mode exposed through exclusive_self_attention=True and its behavior is validated across the relevant attention paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.