huggingface / huggingface/diffusers

[New Pipeline]: Audio-Journey: Visual+LLM-aided Audio Encodec Diffusion

Open
#3,826 3 comments 0 reactions 0 assignees View on GitHub
community-examples New pipeline/model
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Model/Pipeline/Scheduler description

We efficiently trained an Audio Diffusion model with the aid of Alpaca augmented audio captions using AudioSet labels;
[website](https://audiojourney.github.io/)
[preprint](https://github.com/audiojourney/audiojourney.github.io/blob/main/neurIPS_2023_v1.2.pdf)
[Appendix](https://github.com/audiojourney/audiojourney.github.io/blob/main/neurIPS_2023_appendix_v1.3.pdf)
[Implementation](https://github.com/jacksonmichaels/diffusers_with_dataloader)
Weights will be released soon!

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation
@jacksonmichaels

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.