lllyasviel / lllyasviel/stable-diffusion-webui-forge

[Bug]: DORA for Locon and LORA arent loadable "Version Mismatch"

Open
#635 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

Kohaku LORA/LYCORIS modules loaded as LOKR or LOHA trained using dora decomposition method (dora_wd on lycoris) are able to be loaded, but if the module is a vanila LORA or LOCON (Which are treated by the same loader: [locon](https://github.com/KohakuBlueleaf/LyCORIS/blob/main/lycoris/kohya/__init__.py#L34) come up with this error: [error line](https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7/ldm_patched/modules/sd.py#L69)

The primary difference dora is contributing is the addition of a decomposed weight parameter, it seems as though the lycoris module is being used to correctly load these for LOHA/LOKR but not being utilized for LOCON/LORA. I couldn't identify where the exact issue is.

### Steps to reproduce the problem

I've uploaded an example model here to test
This is a lycoris LORA trained using kohya-sd scripts with the following network parameters:

Attempting to use this lora gives the aformentioned version error and does not apply the lora.

### What should have happened?

The lora should be applied.

### What browsers do you use to access the UI ?

_No response_

### Sysinfo

[sysinfo-2024-04-03-13-06.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14852564/sysinfo-2024-04-03-13-06.json)

### Console logs

```Shell
(base) ➜ stable-diffusion-webui-forge git:(main) ✗ ./webui.sh --cuda-stream --pin-shared-memory

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye), Fedora 34+ and openSUSE Leap 15.4 or newer.
################################################################

################################################################
Running on blackroot user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
glibc version is 2.39
Check TCMalloc: libtcmalloc_minimal.so.4
libtcmalloc_minimal.so.4 is linked with libc.so,execute LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so.4
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
Version: f0.0.17v1.8.0rc-latest-276-g29be1da7
Commit hash: 29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7
Legacy Preprocessor init warning: Unable to install insightface automatically. Please try run `pip install insightface` manually.
Launching Web UI with arguments: --cuda-stream --pin-shared-memory
Total VRAM 24238 MB, total RAM 128676 MB
Set vram state to: NORMAL_VRAM
Always pin shared GPU memory
Device: cuda:0 NVIDIA GeForce RTX 3090 Ti : native
Hint: your device supports --cuda-malloc for potential speed improvements.
VAE dtype: torch.bfloat16
CUDA Stream Activated: True
Using pytorch cross attention
ControlNet preprocessor location: /home/blackroot/Desktop/stable-diffusion-webui-forge/models/ControlNetPreprocessor
Loading weights [8ea2b6e4e2] from /home/blackroot/Desktop/stable-diffusion-webui-forge/models/Stable-diffusion/CHEYENNE_v16.safetensors
2024-04-03 07:05:43,361 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL: http://127.0.0.1:7861

To create a public link, set `share=True` in `launch()`.
model_type EPS
UNet ADM Dimension 2816
Startup time: 5.5s (prepare environment: 1.2s, import torch: 1.8s, import gradio: 0.3s, setup paths: 0.4s, other imports: 0.2s, load scripts: 0.7s, create ui: 0.4s, gradio launch: 0.4s).
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['denoiser.sigmas'])
To load target model SDXLClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 9310.80859375
[Memory Management] Model Memory (MB) = 2144.3546981811523
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 6142.453895568848
Moving model(s) has taken 0.20 seconds
Model loaded in 2.9s (load weights from disk: 0.4s, forge load real models: 2.1s, calculate empty prompt: 0.3s).
[LORA] LoRA version mismatch for SDXL: /mnt/ue/loras/dpr1.safetensors
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at ldm_patched/modules/sd.py line 69, where the reported LoRA version mismatch occurs, and compare that path with the linked LyCORIS loader entry point. Reproduce using the linked SD-DORA example model and the console command or sysinfo details. Done means the vanilla LoRA or LoCon DORA model loads and is applied without the version error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.