huggingface / huggingface/diffusers

`set_attention_backend` on one model leaks globally to other models via the active-backend registry

Open
#14,249 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

Calling model.set_attention_backend(...) on a single model does not only configure that model — it also flips a global active backend (_AttentionBackendRegistry.set_active_backend(...)). Any other model in the same process whose per-module attention backend is left unset then silently picks up that global backend inside dispatch_attention_fn (the backend is None fallback path).

This is surprising: setting the backend on model A changes the attention behavior of unrelated model B. It's especially harmful when B calls attention with arguments the leaked backend doesn't support — e.g. a masked attention (attn_mask=...) with a FlashAttention-3 backend, which raises ValueError: attn_mask is not supported for flash-attn 3. Even though the user never asked FA3 to run in B.

Reproduction

import torch
from diffusers.models.attention_dispatch import (
    AttentionBackendName, _AttentionBackendRegistry, dispatch_attention_fn,
)

# Model A opts into a backend; nothing is configured on B.
# set_attention_backend() sets the module's processor._attention_backend AND, as a side
# effect, flips the process-global active backend:
_AttentionBackendRegistry.set_active_backend(AttentionBackendName("_flash_3_hub"))

# Model B just calls dispatch with backend=None (its per-module backend is unset) and a mask:
q = k = v = torch.randn(1, 4, 64, 64, device="cuda", dtype=torch.bfloat16)
mask = torch.zeros(1, 1, 64, 64, device="cuda", dtype=torch.bfloat16)

out = dispatch_attention_fn(q, k, v, attn_mask=mask, backend=None)  # -> uses the leaked global backend
# ValueError: `attn_mask` is not supported for flash-attn 3.

Cc: @dg845 @DN6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing set_attention_backend, _AttentionBackendRegistry.set_active_backend, and dispatch_attention_fn, then run the provided masked-attention reproduction. Determine how an unset per-module backend resolves across models; done means configuring one model no longer changes another model's backend selection, and the masked call does not use the leaked backend.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.