huggingface / huggingface/diffusers
`set_attention_backend` on one model leaks globally to other models via the active-backend registry
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
Calling model.set_attention_backend(...) on a single model does not only configure that model — it also flips a global active backend (_AttentionBackendRegistry.set_active_backend(...)). Any other model in the same process whose per-module attention backend is left unset then silently picks up that global backend inside dispatch_attention_fn (the backend is None fallback path).
This is surprising: setting the backend on model A changes the attention behavior of unrelated model B. It's especially harmful when B calls attention with arguments the leaked backend doesn't support — e.g. a masked attention (attn_mask=...) with a FlashAttention-3 backend, which raises ValueError: attn_mask is not supported for flash-attn 3. Even though the user never asked FA3 to run in B.
Reproduction
import torch
from diffusers.models.attention_dispatch import (
AttentionBackendName, _AttentionBackendRegistry, dispatch_attention_fn,
)
# Model A opts into a backend; nothing is configured on B.
# set_attention_backend() sets the module's processor._attention_backend AND, as a side
# effect, flips the process-global active backend:
_AttentionBackendRegistry.set_active_backend(AttentionBackendName("_flash_3_hub"))
# Model B just calls dispatch with backend=None (its per-module backend is unset) and a mask:
q = k = v = torch.randn(1, 4, 64, 64, device="cuda", dtype=torch.bfloat16)
mask = torch.zeros(1, 1, 64, 64, device="cuda", dtype=torch.bfloat16)
out = dispatch_attention_fn(q, k, v, attn_mask=mask, backend=None) # -> uses the leaked global backend
# ValueError: `attn_mask` is not supported for flash-attn 3.
Cc: @dg845 @DN6
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing set_attention_backend, _AttentionBackendRegistry.set_active_backend, and dispatch_attention_fn, then run the provided masked-attention reproduction. Determine how an unset per-module backend resolves across models; done means configuring one model no longer changes another model's backend selection, and the masked call does not use the leaked backend.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100