kohya-ss / kohya-ss/sd-scripts
Lora training on Playground V2 model
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I got the error when trying to start lora train with Playground V2 model
How to fix it? Looks like an error due model conversion, but I'm not sure.
https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic
Its sdxl but trained from scratch. Works in a1, comfy, fooocus without changin anything.
UPD
Folk tells me that they (Playgorund) have modified TE a bit and that is why i got __Unexpected key__
```
Traceback (most recent call last):
File "S:\kohya_ss-22.3.0\sdxl_train_network.py", line 185, in
trainer.train(args)
File "S:\kohya_ss-22.3.0\train_network.py", line 228, in train
model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "S:\kohya_ss-22.3.0\sdxl_train_network.py", line 44, in load_target_model
) = sdxl_train_util.load_target_model(args, accelerator, sdxl_model_util.MODEL_VERSION_SDXL_BASE_V1_0, weight_dtype)
File "S:\kohya_ss-22.3.0\library\sdxl_train_util.py", line 34, in load_target_model
) = _load_target_model(
File "S:\kohya_ss-22.3.0\library\sdxl_train_util.py", line 75, in _load_target_model
) = sdxl_model_util.load_models_from_sdxl_checkpoint(model_version, name_or_path, device, model_dtype)
File "S:\kohya_ss-22.3.0\library\sdxl_model_util.py", line 270, in load_models_from_sdxl_checkpoint
info2 = _load_state_dict_on_device(text_model2, converted_sd, device=map_location) # remain fp32
File "S:\kohya_ss-22.3.0\library\sdxl_model_util.py", line 158, in _load_state_dict_on_device
raise RuntimeError("Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for CLIPTextModelWithProjection:
Unexpected key(s) in state_dict: "text_projection.weight.weight".
Missing key(s) in state_dict: "text_projection.weight".
Traceback (most recent call last):
File "C:\Users\blackbody\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\blackbody\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "S:\kohya_ss-22.3.0\venv\Scripts\accelerate.exe\__main__.py", line 7, in
File "S:\kohya_ss-22.3.0\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "S:\kohya_ss-22.3.0\venv\lib\site-packages\accelerate\commands\launch.py", line 986, in launch_command
simple_launcher(args)
File "S:\kohya_ss-22.3.0\venv\lib\site-packages\accelerate\commands\launch.py", line 628, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure from the traceback in sdxl_train_network.py, sdxl_train_util.py, and library/sdxl_model_util.py using the linked Playground V2 model. Inspect the state-dict loading path around the reported text_projection.weight.weight and text_projection.weight mismatch. Done means the model loads without this error and LoRA training can start.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100