huggingface / huggingface/diffusers
Integration of ImageBind and StableUnCLIPImg2ImgPipeline for audio2image generation
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
### Model/Pipeline/Scheduler description
For anyone who need, here is a simple demo to illustrate how to integrate ImageBind and StableUnCLIPImg2ImgPipeline for audio2image generation.
### Open source status
- [X] The model implementation is available
- [X] The model weights are available (Only relevant if addition is not a scheduler).
### Provide useful links for the implementation
See also, https://github.com/Zeqiang-Lai/Anything2Image
```python
import imagebind
import torch
from diffusers import StableUnCLIPImg2ImgPipeline
# construct models
device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipe = StableUnCLIPImg2ImgPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-unclip", torch_dtype=torch.float16, variation="fp16"
)
pipe = pipe.to(device)
model = imagebind.imagebind_huge(pretrained=True)
model.eval()
model.to(device)
# generate image
with torch.no_grad():
audio_paths=["assets/wav/bird_audio.wav"]
embeddings = model.forward({
imagebind.ModalityType.AUDIO: imagebind.load_and_transform_audio_data(audio_paths, device),
})
embeddings = embeddings[imagebind.ModalityType.AUDIO]
images = pipe(image_embeds=embeddings.half()).images
images[0].save("bird_audio.png")
```
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.