kohya-ss / kohya-ss/sd-scripts
Training failed for unknown reason
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I get an exception after trying to train my first model (v1.5). I have low vram ~8gb so I tried to run with the lowest settings possible and followed https://www.youtube.com/watch?v=70H03cv57-o for the most part except that I set mixed-precision to no since that gave me another exception ("enable xformers for u-net").
Does someone know what I did wrong?
I can't make out any specific errorlog that would guide me into the right direction...
thanks in advance
```
You have disabled the safety checker for by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
UNet2DConditionModel: 64, 8, 768, False, False
U-Net converted to original U-Net
Enable memory efficient attention for U-Net
[Dataset 0]
caching latents.
checking cache validity...
100%|██████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:00 True
CrossAttnDownBlock2D False -> True
CrossAttnDownBlock2D False -> True
DownBlock2D False -> True
UNetMidBlock2DCrossAttn False -> True
UpBlock2D False -> True
CrossAttnUpBlock2D False -> True
CrossAttnUpBlock2D False -> True
CrossAttnUpBlock2D False -> True
prepare optimizer, data loader etc.
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA SETUP: Loading binary C:\Users\XXX\AI\kohya_ss\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...
use 8-bit AdamW optimizer | {}
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 3700
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 3700
num epochs / epoch数: 1
batch size per device / バッチサイズ: 1
total train batch size (with parallel & distributed & accumulation) / 総バッチサイズ(並列学習、勾配合計含む): 1
gradient ccumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 3700
steps: 0%| | 0/3700 [00:00:7 │
│ │
│ 4 from accelerate.commands.accelerate_cli import main │
│ 5 if __name__ == '__main__': │
│ 6 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ ❱ 7 │ sys.exit(main()) │
│ 8 │
│ │
│ C:\Users\XXX\AI\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py:45 in │
│ main │
│ │
│ 42 │ │ exit(1) │
│ 43 │ │
│ 44 │ # Run │
│ ❱ 45 │ args.func(args) │
│ 46 │
│ 47 │
│ 48 if __name__ == "__main__": │
│ │
│ C:\Users\XXX\AI\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py:918 in │
│ launch_command │
│ │
│ 915 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │
│ 916 │ │ sagemaker_launcher(defaults, args) │
│ 917 │ else: │
│ ❱ 918 │ │ simple_launcher(args) │
│ 919 │
│ 920 │
│ 921 def main(): │
│ │
│ C:\Users\XXX\AI\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py:580 in │
│ simple_launcher │
│ │
│ 577 │ process.wait() │
│ 578 │ if process.returncode != 0: │
│ 579 │ │ if not args.quiet: │
│ ❱ 580 │ │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │
│ 581 │ │ else: │
│ 582 │ │ │ sys.exit(1) │
│ 583 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['C:\\Users\\XXX\\AI\\kohya_ss\\venv\\Scripts\\python.exe', './train_db.py',
'--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5',
'--train_data_dir=C:/Users/.../img', '--resolution=512,512',
'--output_dir=C:/Users/.../model',
'--logging_dir=C:/Users/.../log', '--save_model_as=safetensors', '--output_name=test',
'--lr_scheduler_num_cycles=1', '--max_data_loader_n_workers=1', '--learning_rate=0.0001', '--lr_scheduler=constant',
'--train_batch_size=1', '--max_train_steps=3700', '--save_every_n_epochs=1', '--mixed_precision=no',
'--save_precision=fp16', '--seed=1234', '--caption_extension=.txt', '--cache_latents', '--optimizer_type=AdamW8bit',
'--max_data_loader_n_workers=1', '--clip_skip=2', '--bucket_reso_steps=64', '--mem_eff_attn',
'--gradient_checkpointing', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' returned non-zero exit status
3221226356.
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported train_db.py launch and the bitsandbytes and accelerate output, then reproduce the Windows training command with the provided settings. Check the child-process error around the first training step and the exit status 3221226356. Done means identifying a reproducible cause and documenting an actionable error or fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100