huggingface / huggingface/diffusers

Suggestion to add TEs

Open
#10,890 2 comments 3 reactions 0 assignees View on GitHub
remote-vae wip
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Did you like the remote VAE solution?

I do... it's helpful when training/finetuning models, makes the infrastructure easier to set up.

### What can be improved about the current solution?

Adding TEs would be very helpful for a complete solution for auxiliary model off-loading during training.
Consider caching, then it could further simplify model fine-tuning setups.

### What other VAEs you would like to see if the pilot goes well?

StepVideo VAE: https://huggingface.co/stepfun-ai/stepvideo-t2v/tree/main/vae
HunYuan CLIP (for Stepvideo): https://huggingface.co/stepfun-ai/stepvideo-t2v/tree/main/hunyuan_clip
TE for LuminaVision2: [Gemma-2-2B](https://huggingface.co/google/gemma-2-2b)

### Notify the members of the team

@hlky @sayakpaul

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.