kohya-ss / kohya-ss/sd-scripts

sdxl_merge_lora.py does not work properly

Open
#972 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

In networks/sdxl_merge_lora.py line 80
`if module_name not in name_to_module:`
The **module_name** is always not in **name_to_module**
because the different name between unet and diffuser format
**name_to_module** contains something like **lora_unet_input_blocks_1_0_in_layers_2**
**module_name** always like **lora_unet_down_blocks_0_resnets_0_conv1**

It should have a convertion or mapping between different name.

I made a .json file for mapping but I think it might not the best solution.
[diffusers_keys_mapping.json](https://github.com/kohya-ss/sd-scripts/files/13467571/diffusers_keys_mapping.json)

My workaround:
```
...
diffusers_keys = {}
script_dir = os.path.dirname(__file__)
mapping_file_path = os.path.join(script_dir, "diffusers_keys_mapping.json")
# open diffusers_keys.json file
with open(mapping_file_path, "r") as f:
diffusers_keys = json.load(f)
f.close()
for model, ratio in zip(models, ratios):
print(f"loading: {model}")
lora_sd, _ = load_state_dict(model, merge_dtype)

print(f"merging...")
for key in tqdm(lora_sd.keys()):
if "lora_down" in key:
up_key = key.replace("lora_down", "lora_up")
alpha_key = key[: key.index("lora_down")] + "alpha"

# find original module for this lora
module_name = ".".join(key.split(".")[:-2]) # remove trailing ".lora_down.weight"
patch_module_name = diffusers_keys[module_name]
# print(f'debug key:{key}')
print(f'debug patch_module_name:{patch_module_name}')
if patch_module_name not in name_to_module:
print(f"no module found for LoRA weight: {key}")
...
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at networks/sdxl_merge_lora.py line 80 and compare the unet and diffuser-format names described in the issue. Review the proposed diffusers_keys_mapping.json workaround, then verify that SDXL LoRA modules are found during merging without the reported name mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.