huggingface / huggingface/diffusers

Integration of ImageBind and StableUnCLIPImg2ImgPipeline for audio2image generation

Offen
#3,444 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
community-examples New pipeline/model
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Model/Pipeline/Scheduler description

For anyone who need, here is a simple demo to illustrate how to integrate ImageBind and StableUnCLIPImg2ImgPipeline for audio2image generation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

See also, https://github.com/Zeqiang-Lai/Anything2Image

```python
import imagebind
import torch
from diffusers import StableUnCLIPImg2ImgPipeline

# construct models
device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipe = StableUnCLIPImg2ImgPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-unclip", torch_dtype=torch.float16, variation="fp16"
)
pipe = pipe.to(device)

model = imagebind.imagebind_huge(pretrained=True)
model.eval()
model.to(device)

# generate image
with torch.no_grad():
audio_paths=["assets/wav/bird_audio.wav"]
embeddings = model.forward({
imagebind.ModalityType.AUDIO: imagebind.load_and_transform_audio_data(audio_paths, device),
})
embeddings = embeddings[imagebind.ModalityType.AUDIO]
images = pipe(image_embeds=embeddings.half()).images
images[0].save("bird_audio.png")
```

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by reviewing the linked Anything2Image repository and the provided ImageBind and StableUnCLIPImg2ImgPipeline demo. Identify the appropriate integration point in diffusers and confirm the expected model, weights, API, tests, and documentation before proceeding; done means the supported audio-to-image integration is implemented and validated.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
ai, machine-learning
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.