huggingface / huggingface/diffusers

Let's make `UNet2DConditionModel` take in not only class labels, but also continuous labels

Open
#10,257 1 comment 0 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

The current `UNet2DConditoinModel` can take in class labels as described [here](https://huggingface.co/docs/diffusers/v0.31.0/en/api/models/unet2d-cond#diffusers.UNet2DConditionModel.num_class_embeds).

However, in my case and also some other people are using it with **continuous labels** and in that case there needs to be `nn.Linear`-like layer in addition to `nn.Embedding` in [about this place](https://github.com/huggingface/diffusers/blob/89e4d6219805975bd7d253a267e1951badc9f1c0/src/diffusers/models/unets/unet_2d_condition.py#L616)

As I mentioned, there seems to be some needs for it, and even if the paper does not address the issue in the way I described, [this paper](https://arxiv.org/abs/2405.03546) (Continuous Conditional Diffusion Models) is one of the trials to use continuous labels in diffusion models.

To be clearer, I'm trying to generate images based on conditions that are regression-like values, such as heights and widths

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.