kohya-ss / kohya-ss/sd-scripts

Help solve the problem! When starting the trainer it gives an error, I do everything through Google Colab

Open
#1,198 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Help solve the problem! When starting the trainer it gives an error, I do everything through Google Colab

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /content/kohya-trainer/sdxl_train_network.py:3 in │
│ │
│ 1 import argparse │
│ 2 import torch │
│ ❱ 3 from library import sdxl_model_util, sdxl_train_util, train_util │
│ 4 import train_network │
│ 5 │
│ 6 │
│ │
│ /content/kohya-trainer/library/sdxl_train_util.py:10 in │
│ │
│ 7 from accelerate import init_empty_weights │
│ 8 from tqdm import tqdm │
│ 9 from transformers import CLIPTokenizer │
│ ❱ 10 from library import model_util, sdxl_model_util, train_util, sdxl_original_unet │
│ 11 from library.sdxl_lpw_stable_diffusion import SdxlStableDiffusionLongPromptWeightingPipe │
│ 12 │
│ 13 TOKENIZER1_PATH = "openai/clip-vit-large-patch14" │
│ │
│ /content/kohya-trainer/library/train_util.py:36 in │
│ │
│ 33 import torch │
│ 34 from torch.nn.parallel import DistributedDataParallel as DDP │
│ 35 from torch.optim import Optimizer │
│ ❱ 36 from torchvision import transforms │
│ 37 from transformers import CLIPTokenizer │
│ 38 import transformers │
│ 39 from diffusers.optimization import SchedulerType, TYPE_TO_SCHEDULER_FUNCTION │
│ │
│ /usr/local/lib/python3.10/dist-packages/torchvision/__init__.py:6 in │
│ │
│ 3 from modulefinder import Module │
│ 4 │
│ 5 import torch │
│ ❱ 6 from torchvision import _meta_registrations, datasets, io, models, ops, transforms, util │
│ 7 │
│ 8 from .extension import _HAS_OPS │
│ 9 │
│ │
│ /usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py:164 in │
│ │
│ 161 │
│ 162 │
│ 163 @torch._custom_ops.impl_abstract("torchvision::nms") │
│ ❱ 164 def meta_nms(dets, scores, iou_threshold): │
│ 165 │ torch._check(dets.dim() == 2, lambda: f"boxes should be a 2d tensor, got {dets.dim() │
│ 166 │ torch._check(dets.size(1) == 4, lambda: f"boxes should have 4 elements in dimension │
│ 167 │ torch._check(scores.dim() == 1, lambda: f"scores should be a 1d tensor, got {scores. │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_ops.py:253 in inner │
│ │
│ 250 │ """ │
│ 251 │ │
│ 252 │ def inner(func): │
│ ❱ 253 │ │ custom_op = _find_custom_op(qualname, also_check_torch_library=True) │
│ 254 │ │ custom_op.impl_abstract(_stacklevel=3)(func) │
│ 255 │ │ return func │
│ 256 │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1076 in _find_custom_op │
│ │
│ 1073 │ │ raise RuntimeError( │
│ 1074 │ │ │ f"Could not find custom op \"{qualname}\". Did you register it via " │
│ 1075 │ │ │ f"the torch._custom_ops API?") │
│ ❱ 1076 │ overload = get_op(qualname) │
│ 1077 │ result = custom_op_from_existing(overload) │
│ 1078 │ return result │
│ 1079 │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1062 in get_op │
│ │
│ 1059 │ │ error_not_found() │
│ 1060 │ opnamespace = getattr(torch.ops, ns) │
│ 1061 │ if not hasattr(opnamespace, name): │
│ ❱ 1062 │ │ error_not_found() │
│ 1063 │ packet = getattr(opnamespace, name) │
│ 1064 │ if not hasattr(packet, 'default'): │
│ 1065 │ │ error_not_found() │
│ │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1052 in error_not_found │
│ │
│ 1049 │
│ 1050 def get_op(qualname): │
│ 1051 │ def error_not_found(): │
│ ❱ 1052 │ │ raise ValueError( │
│ 1053 │ │ │ f"Could not find the operator {qualname}. Please make sure you have " │
│ 1054 │ │ │ f"already registered the operator and (if registered from C++) " │
│ 1055 │ │ │ f"loaded it via torch.ops.load_library.") │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Could not find the operator torchvision::nms. Please make sure you have already
registered the operator and (if registered from C++) loaded it via torch.ops.load_library.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/bin/accelerate:8 in │
│ │
│ 5 from accelerate.commands.accelerate_cli import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py:45 in main │
│ │
│ 42 │ │ exit(1) │
│ 43 │ │
│ 44 │ # Run │
│ ❱ 45 │ args.func(args) │
│ 46 │
│ 47 │
│ 48 if __name__ == "__main__": │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:918 in launch_command │
│ │
│ 915 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │
│ 916 │ │ sagemaker_launcher(defaults, args) │
│ 917 │ else: │
│ ❱ 918 │ │ simple_launcher(args) │
│ 919 │
│ 920 │
│ 921 def main(): │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:580 in simple_launcher │
│ │
│ 577 │ process.wait() │
│ 578 │ if process.returncode != 0: │
│ 579 │ │ if not args.quiet: │
│ ❱ 580 │ │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │
│ 581 │ │ else: │
│ 582 │ │ │ sys.exit(1) │
│ 583 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/usr/bin/python3', 'sdxl_train_network.py',
'--sample_prompts=/content/LoRA/config/sample_prompt.toml',
'--config_file=/content/LoRA/config/config_file.toml']' returned non-zero exit status 1.

Contributor guide

No contributing guide indexed for this repository

Research direction

The failure occurs while starting sdxl_train_network.py, through library/sdxl_train_util.py and library/train_util.py, when importing torchvision.transforms. Start by reproducing the Google Colab startup and checking the installed PyTorch and torchvision environment around the missing torchvision::nms operator. Done means the trainer imports and starts without this traceback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.