Project-MONAI / Project-MONAI/tutorials

Clarification on mutually exclusive use of self-attention and cross-attention in DiffusionModelUNetMaisi

Open
#1,993 4 comments 0 reactions 3 assignees View on GitHub

@dongyang0122 is already working on this.

Since Jun 9, 2025.

Dominant language
Jupyter Notebook
Stars
2.5k
Forks
803
Avg merge
6d 22h
Merged PRs (30d)
3

Description

Hi MONAI team,

while reading through the implementation of DiffusionModelUNetMaisi, I noticed the following logic for enabling attention at each level:

with_attn = attention_levels[i] and not with_conditioning
with_cross_attn = attention_levels[i] and with_conditioning

This effectively means that self-attention is never used when the model is in conditioning mode (with_conditioning=True), even if attention_levels[i] is True.

Is this behavior intentional?

In other diffusion-based architectures such as Stable Diffusion, it is common practice to enable both self-attention and cross-attention simultaneously within the same layers.

Would it be acceptable, or even recommended, to modify the logic as follows to allow both mechanisms in parallel?
Or is there a specific reason this mutual exclusivity was enforced?

Looking forward to your insights, and thank you for the great work on this model!

Best regards,
Daniele Molino

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.