lllyasviel / lllyasviel/stable-diffusion-webui-forge

Add option to keep model in VRAM instead of unloading it after each generation

Open
#1,245 13 comments 9 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I am running the FP16 version of Flux and the fp16 T5 text encoder on my RTX 2060 laptop with 32 GB RAM. I was surprised to see WebUI forge having faster speeds by multiple magnitudes compared to Comfy (11 minutes vs 2 minutes), so great job on the optimization here, @lllyasviel !

However, running it in FP16 is really tight on my RAM as well so loading parts of the model into VRAM takes quite a bit of time. When I press generation, moving models adds around 1 minute to the generation time.

So it would be really cool to have an option to turn this behavior off. Once loaded, the model should stay in VRAM until I close the program. This way there's be no moving models process between generations and it would speed up the experience up by a lot. Please consider it.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Trace the model-loading and unloading flow around generation, then identify where a configurable keep-in-VRAM option would fit. Done means models remain in VRAM between generations when enabled and generation still works normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.