kohya-ss / kohya-ss/sd-scripts
SDXL trainning ,saving checkpoint error. thank you
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
aN found in latents, replacing with zeros
steps: 17%|████████▋ | 1690/10140 [1:06:44<5:33:42, 2.37s/it, loss=0.00321]
saving checkpoint: ./output\kincoraXL-000001.safetensors
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ E:\lora-scripts\sd-scripts\sdxl_train_network.py:174 in │
│ │
│ 171 │ args = train_util.read_config_from_file(args, parser) │
│ 172 │ │
│ 173 │ trainer = SdxlNetworkTrainer() │
│ ❱ 174 │ trainer.train(args) │
│ 175 │
│ │
│ E:\lora-scripts\sd-scripts\train_network.py:869 in train │
│ │
│ 866 │ │ │ │ saving = (epoch + 1) % args.save_every_n_epochs == 0 and (epoch + 1) < n │
│ 867 │ │ │ │ if is_main_process and saving: │
│ 868 │ │ │ │ │ ckpt_name = train_util.get_epoch_ckpt_name(args, "." + args.save_mod │
│ ❱ 869 │ │ │ │ │ save_model(ckpt_name, accelerator.unwrap_model(network), global_step │
│ 870 │ │ │ │ │ │
│ 871 │ │ │ │ │ remove_epoch_no = train_util.get_remove_epoch_no(args, epoch + 1) │
│ 872 │ │ │ │ │ if remove_epoch_no is not None: │
│ │
│ E:\lora-scripts\sd-scripts\train_network.py:711 in save_model │
│ │
│ 708 │ │ │ sai_metadata = train_util.get_sai_model_spec(None, args, self.is_sdxl, True, │
│ 709 │ │ │ metadata_to_save.update(sai_metadata) │
│ 710 │ │ │ │
│ ❱ 711 │ │ │ unwrapped_nw.save_weights(ckpt_file, save_dtype, metadata_to_save) │
│ 712 │ │ │ if args.huggingface_repo_id is not None: │
│ 713 │ │ │ │ huggingface_util.upload(args, ckpt_file, "/" + ckpt_name, force_sync_upl │
│ 714 │
│ │
│ E:\lora-scripts\sd-scripts\networks\lora.py:1095 in save_weights │
│ │
│ 1092 │ │ │ metadata["sshs_model_hash"] = model_hash │
│ 1093 │ │ │ metadata["sshs_legacy_hash"] = legacy_hash │
│ 1094 │ │ │ │
│ ❱ 1095 │ │ │ save_file(state_dict, file, metadata) │
│ 1096 │ │ else: │
│ 1097 │ │ │ torch.save(state_dict, file) │
│ 1098 │
│ │
│ E:\lora-scripts\venv\lib\site-packages\safetensors\torch.py:232 in save_file │
│ │
│ 229 │ save_file(tensors, "model.safetensors") │
│ 230 │ ``` │
│ 231 │ """ │
│ ❱ 232 │ serialize_file(_flatten(tensors), filename, metadata=metadata) │
│ 233 │
│ 234 │
│ 235 def load_file(filename: Union[str, os.PathLike], device="cpu") -> Dict[str, torch.Tensor │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: argument 'metadata': 'int' object cannot be converted to 'PyString'
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at train_network.py:869 and save_model, then follow networks/lora.py:1095 into the safetensors save call. Reproduce the checkpoint-saving failure and inspect the metadata passed there, especially the value reported as an integer. Done means SDXL training can save the checkpoint without the metadata TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100