huggingface / huggingface/diffusers

Support training where width does not equal height for Qwen-Image

Open
#12,102 2 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

To support training where width does not equal height for Qwen-Image, the following code:

```
img_shapes = [
(1, args.resolution // vae_scale_factor // 2, args.resolution // vae_scale_factor // 2)
] * bsz
noisy_model_input = noisy_model_input.permute(0, 2, 1, 3, 4)
packed_noisy_model_input = QwenImagePipeline._pack_latents(
noisy_model_input,
batch_size=model_input.shape[0],
num_channels_latents=model_input.shape[1],
height=model_input.shape[3],
width=model_input.shape[4],
)
model_pred = transformer(
hidden_states=packed_noisy_model_input,
encoder_hidden_states=prompt_embeds,
encoder_hidden_states_mask=prompt_embeds_mask,
timestep=timesteps / 1000,
img_shapes=img_shapes,
txt_seq_lens=prompt_embeds_mask.sum(dim=1).tolist(),
return_dict=False,
)[0]
model_pred = QwenImagePipeline._unpack_latents(
model_pred, args.resolution, args.resolution, vae_scale_factor
)
```
should be modified to:
```
img_shapes = [
(1, model_input.shape[3] // 2, model_input.shape[4] // 2)
] * bsz
noisy_model_input = noisy_model_input.permute(0, 2, 1, 3, 4)
packed_noisy_model_input = QwenImagePipeline._pack_latents(
noisy_model_input,
batch_size=model_input.shape[0],
num_channels_latents=model_input.shape[1],
height=model_input.shape[3],
width=model_input.shape[4],
)
model_pred = transformer(
hidden_states=packed_noisy_model_input,
encoder_hidden_states=prompt_embeds,
encoder_hidden_states_mask=prompt_embeds_mask,
timestep=timesteps / 1000,
img_shapes=img_shapes,
txt_seq_lens=prompt_embeds_mask.sum(dim=1).tolist(),
return_dict=False,
)[0]
model_pred = QwenImagePipeline._unpack_latents(
model_pred, model_input.shape[3] * vae_scale_factor, model_input.shape[4] * vae_scale_factor, vae_scale_factor
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.