huggingface / huggingface/diffusers
Integration of ImageBind and StableUnCLIPImg2ImgPipeline for audio2image generation
- Vorherrschende Sprache
- Python
- Sterne
- 34.5k
- Forks
- 7.3k
- Ø Merge
- 3 T. 3 Std.
- Gemergte PRs (30 T.)
- 91
Beschreibung
### Model/Pipeline/Scheduler description
For anyone who need, here is a simple demo to illustrate how to integrate ImageBind and StableUnCLIPImg2ImgPipeline for audio2image generation.
### Open source status
- [X] The model implementation is available
- [X] The model weights are available (Only relevant if addition is not a scheduler).
### Provide useful links for the implementation
See also, https://github.com/Zeqiang-Lai/Anything2Image
```python
import imagebind
import torch
from diffusers import StableUnCLIPImg2ImgPipeline
# construct models
device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipe = StableUnCLIPImg2ImgPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-unclip", torch_dtype=torch.float16, variation="fp16"
)
pipe = pipe.to(device)
model = imagebind.imagebind_huge(pretrained=True)
model.eval()
model.to(device)
# generate image
with torch.no_grad():
audio_paths=["assets/wav/bird_audio.wav"]
embeddings = model.forward({
imagebind.ModalityType.AUDIO: imagebind.load_and_transform_audio_data(audio_paths, device),
})
embeddings = embeddings[imagebind.ModalityType.AUDIO]
images = pipe(image_embeds=embeddings.half()).images
images[0].save("bird_audio.png")
```
Beitragsleitfaden
Rechercherichtung
Start by reviewing the linked Anything2Image repository and the provided ImageBind and StableUnCLIPImg2ImgPipeline demo. Identify the appropriate integration point in diffusers and confirm the expected model, weights, API, tests, and documentation before proceeding; done means the supported audio-to-image integration is implemented and validated.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python, pytorch
- Bereich
- ai, machine-learning
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 25/100