kohya-ss / kohya-ss/sd-scripts

SDXL trainning ,saving checkpoint error. thank you

Open
#764 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

aN found in latents, replacing with zeros
steps: 17%|████████▋ | 1690/10140 [1:06:44<5:33:42, 2.37s/it, loss=0.00321]
saving checkpoint: ./output\kincoraXL-000001.safetensors
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ E:\lora-scripts\sd-scripts\sdxl_train_network.py:174 in │
│ │
│ 171 │ args = train_util.read_config_from_file(args, parser) │
│ 172 │ │
│ 173 │ trainer = SdxlNetworkTrainer() │
│ ❱ 174 │ trainer.train(args) │
│ 175 │
│ │
│ E:\lora-scripts\sd-scripts\train_network.py:869 in train │
│ │
│ 866 │ │ │ │ saving = (epoch + 1) % args.save_every_n_epochs == 0 and (epoch + 1) < n │
│ 867 │ │ │ │ if is_main_process and saving: │
│ 868 │ │ │ │ │ ckpt_name = train_util.get_epoch_ckpt_name(args, "." + args.save_mod │
│ ❱ 869 │ │ │ │ │ save_model(ckpt_name, accelerator.unwrap_model(network), global_step │
│ 870 │ │ │ │ │ │
│ 871 │ │ │ │ │ remove_epoch_no = train_util.get_remove_epoch_no(args, epoch + 1) │
│ 872 │ │ │ │ │ if remove_epoch_no is not None: │
│ │
│ E:\lora-scripts\sd-scripts\train_network.py:711 in save_model │
│ │
│ 708 │ │ │ sai_metadata = train_util.get_sai_model_spec(None, args, self.is_sdxl, True, │
│ 709 │ │ │ metadata_to_save.update(sai_metadata) │
│ 710 │ │ │ │
│ ❱ 711 │ │ │ unwrapped_nw.save_weights(ckpt_file, save_dtype, metadata_to_save) │
│ 712 │ │ │ if args.huggingface_repo_id is not None: │
│ 713 │ │ │ │ huggingface_util.upload(args, ckpt_file, "/" + ckpt_name, force_sync_upl │
│ 714 │
│ │
│ E:\lora-scripts\sd-scripts\networks\lora.py:1095 in save_weights │
│ │
│ 1092 │ │ │ metadata["sshs_model_hash"] = model_hash │
│ 1093 │ │ │ metadata["sshs_legacy_hash"] = legacy_hash │
│ 1094 │ │ │ │
│ ❱ 1095 │ │ │ save_file(state_dict, file, metadata) │
│ 1096 │ │ else: │
│ 1097 │ │ │ torch.save(state_dict, file) │
│ 1098 │
│ │
│ E:\lora-scripts\venv\lib\site-packages\safetensors\torch.py:232 in save_file │
│ │
│ 229 │ save_file(tensors, "model.safetensors") │
│ 230 │ ``` │
│ 231 │ """ │
│ ❱ 232 │ serialize_file(_flatten(tensors), filename, metadata=metadata) │
│ 233 │
│ 234 │
│ 235 def load_file(filename: Union[str, os.PathLike], device="cpu") -> Dict[str, torch.Tensor │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: argument 'metadata': 'int' object cannot be converted to 'PyString'

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at train_network.py:869 and save_model, then follow networks/lora.py:1095 into the safetensors save call. Reproduce the checkpoint-saving failure and inspect the metadata passed there, especially the value reported as an integer. Done means SDXL training can save the checkpoint without the metadata TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.