lllyasviel / lllyasviel/stable-diffusion-webui-forge

Issues with cuda out of memory

Open
#2,968 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I have cuda out of memory on 4070 super

I've worked with this ui for year and it was fine
But after some updates (I don't understand exactly) I started to watch this problem and log didn't help me well

To create a public link, set `share=True` in `launch()`.
Startup time: 14.5s (prepare environment: 2.4s, launcher: 0.4s, import torch: 6.1s, initialize shared: 0.1s, other imports: 0.2s, load scripts: 1.6s, create ui: 2.1s, gradio launch: 1.5s).
Environment vars changed: {'stream': True, 'inference_memory': 1024.0, 'pin_shared_memory': False}
[GPU Setting] You will use 91.66% GPU memory (11257.00 MB) to load weights, and use 8.34% GPU memory (1024.00 MB) to do matrix computation.
Environment vars changed: {'stream': False, 'inference_memory': 1024.0, 'pin_shared_memory': False}
[GPU Setting] You will use 91.66% GPU memory (11257.00 MB) to load weights, and use 8.34% GPU memory (1024.00 MB) to do matrix computation.
Environment vars changed: {'stream': True, 'inference_memory': 1024.0, 'pin_shared_memory': False}
[GPU Setting] You will use 91.66% GPU memory (11257.00 MB) to load weights, and use 8.34% GPU memory (1024.00 MB) to do matrix computation.
Loading Model: {'checkpoint_info': {'filename': 'D:\\Forge\\webui\\models\\Stable-diffusion\\flux1-dev-bnb-nf4-v2.safetensors', 'hash': 'f0770152'}, 'additional_modules': [], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
StateDict Keys: {'transformer': 1722, 'vae': 244, 'text_encoder': 198, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Using Detected UNet Type: nf4
Using pre-quant state dict!
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': 'nf4', 'computation_dtype': torch.bfloat16}
Model loaded in 0.7s (unload existing model: 0.2s, forge model load: 0.5s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
[Unload] Trying to free 7725.00 MB for cuda:0 with 0 models keep loaded ... Done.
[Memory Management] Target: JointTextEncoder, Free GPU: 11041.00 MB, Model Require: 5154.62 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 4862.38 MB, All loaded to GPU.
Moving model(s) has taken 2.46 seconds
Traceback (most recent call last):
  File "D:\Forge\webui\modules_forge\main_thread.py", line 30, in work
    self.result = self.func(*self.args, **self.kwargs)
  File "D:\Forge\webui\modules\txt2img.py", line 131, in txt2img_function
    processed = processing.process_images(p)
  File "D:\Forge\webui\modules\processing.py", line 842, in process_images
    res = process_images_inner(p)
  File "D:\Forge\webui\modules\processing.py", line 962, in process_images_inner
    p.setup_conds()
  File "D:\Forge\webui\modules\processing.py", line 1601, in setup_conds
    super().setup_conds()
  File "D:\Forge\webui\modules\processing.py", line 505, in setup_conds
    self.c = self.get_conds_with_caching(prompt_parser.get_multicond_learned_conditioning, prompts, total_steps, [self.cached_c], self.extra_network_data)
  File "D:\Forge\webui\modules\processing.py", line 474, in get_conds_with_caching
    cache[1] = function(shared.sd_model, required_prompts, steps, hires_steps, shared.opts.use_old_scheduling)
  File "D:\Forge\webui\modules\prompt_parser.py", line 262, in get_multicond_learned_conditioning
    learned_conditioning = get_learned_conditioning(model, prompt_flat_list, steps, hires_steps, use_old_scheduling)
  File "D:\Forge\webui\modules\prompt_parser.py", line 189, in get_learned_conditioning
    conds = model.get_learned_conditioning(texts)
  File "D:\Forge\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\Forge\webui\backend\diffusion_engine\flux.py", line 87, in get_learned_conditioning
    cond_t5 = self.text_processing_engine_t5(prompt)
  File "D:\Forge\webui\backend\text_processing\t5_engine.py", line 139, in __call__
    z = self.process_tokens([tokens], [multipliers])[0]
  File "D:\Forge\webui\backend\text_processing\t5_engine.py", line 150, in process_tokens
    z = self.encode_with_transformers(tokens)
  File "D:\Forge\webui\backend\text_processing\t5_engine.py", line 62, in encode_with_transformers
    z = self.text_encoder(
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Forge\webui\backend\nn\t5.py", line 207, in forward
    return self.encoder(x, *args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Forge\webui\backend\nn\t5.py", line 188, in forward
    x, past_bias = l(x, mask, past_bias)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Forge\webui\backend\nn\t5.py", line 164, in forward
    x, past_bias = self.layer[0](x, mask, past_bias)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Forge\webui\backend\nn\t5.py", line 151, in forward
    output, past_bias = self.SelfAttention(self.layer_norm(x), mask=mask, past_bias=past_bias)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Forge\webui\backend\nn\t5.py", line 129, in forward
    k = self.k(x)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Forge\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Forge\webui\backend\operations.py", line 151, in forward
    weight, bias, signal = weights_manual_cast(self, x)
  File "D:\Forge\webui\backend\operations.py", line 79, in weights_manual_cast
    weight, bias = get_weight_and_bias(layer, weight_args, bias_args, weight_fn=weight_fn, bias_fn=bias_fn)
  File "D:\Forge\webui\backend\operations.py", line 35, in get_weight_and_bias
    weight = weight.to(**weight_args)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB. GPU
CUDA out of memory. Tried to allocate 64.00 MiB. GPU

What I can do with this?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the FLUX generation failure on the reported 4070 Super using the traceback entry points in backend/text_processing/t5_engine.py and backend/operations.py. Trace the failed 64 MiB allocation through backend/nn/t5.py and compare it with the reported GPU memory settings; done means generation completes without the CUDA out-of-memory error and the relevant behavior is covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.