huggingface / huggingface/diffusers

Support MuLan, a plug-and-play language adapter to adapt existing diffusion model for up to 110+ languages without additional training

未关闭
#7,948 2 条评论 3 个 reaction 已指派 0 人 在 GitHub 查看
community-examples contributions-welcome stale
主要语言
Python
星标
34.5k
派生
7.3k
平均合并
3 天 3 小时
30 天内合并 PR
91

描述

### Model/Pipeline/Scheduler description

```diff
# pip install mulankit
from diffusers import StableDiffusionPipeline
+ import mulankit

pipe = StableDiffusionPipeline.from_pretrained('Lykon/dreamshaper-8')
+ pipe = mulankit.transform(pipe, 'mulanai/mulan-lang-adapter::sd15_aesthetic.pth')
image = pipe('一只蓝色的🐶 in the 바다').images[0]
```

|一只蓝色的 🐶 in the 바다 (Dreamshaper-8)| レゴシュワルツェネッガー (SDXL-lightning)| 一只可爱的猫头鹰 (MVDream) | 海浪风景 (AnimateDiff) |
|--- | ---| --- | --- |
| ![image](https://github.com/huggingface/diffusers/assets/26198430/b17cc59c-c05d-45fe-bbb5-6608b42d11b4) | ![image](https://github.com/huggingface/diffusers/assets/26198430/9c179e96-214f-4118-8cc8-997c4dbfdec4) | ![image](https://github.com/huggingface/diffusers/assets/26198430/e54521d8-db8f-4603-b7be-4542ab77d11f) | ![image](https://github.com/huggingface/diffusers/assets/26198430/f526469b-b886-4051-b37f-558703dba631)|

MuLan supports

- Base models: Stable Diffusion 1.5, 2.1, XL, Pixart-Alpha/Sigma.
- Downstream models: ControlNet, LCM, LoRA, finetuned models and etc.
- Video models: AnimateDiff.
- 3D models: MVDream.

### Open source status

- [X] The model implementation is available.
- [X] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

https://github.com/mulanai/MuLan
https://huggingface.co/mulanai/mulan-lang-adapter

贡献指南

打开贡献指南

调研方向

Start with the StableDiffusionPipeline example and review the MuLan implementation at https://github.com/mulanai/MuLan and its weights at Hugging Face. Determine how support should cover the listed base, downstream, video, and 3D models. Done means MuLan can be integrated through diffusers for the described multilingual generation cases.

由索引模型根据 Issue 内容生成。

评估

技术栈
python, pytorch
领域
ai, machine-learning
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。