huggingface / huggingface/diffusers

Full support for Flux attention masking

Open
#10,194 2 comments 4 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
SimpleTuner/Kohya allow T5 attention masked training, however this is not currently supported natively in diffusers

**Describe the solution you'd like.**
Already implemented and used in Simpletuner and Kohya: https://github.com/bghira/SimpleTuner/blob/main/helpers/models/flux/transformer.py

**Describe alternatives you've considered.**
Recent implementation doesn't really solve the use case of using existing fine tunes with attention masking with diffusers
https://github.com/huggingface/diffusers/pull/10122

**Additional context.**
@yiyixuxu @bghira @AmericanPresidentJimmyCarter

@bghira's suggestion: "i'd suggested they add encoder_attention_mask and image_attention_mask and if image_attention_mask is None that they could then 1-fill those positions and just cat them together"

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.