kohya-ss / kohya-ss/sd-scripts

Random crashes: 'NoneType' object has no attribute 'ndim'

Open
#1,577 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Training is intermittently crashing on some sample generations. Error:

[rank3]: Traceback (most recent call last):
[rank3]: File "/root/run/kohya/flux_train.py", line 905, in
[rank3]: train(args)
[rank3]: File "/root/run/kohya/flux_train.py", line 760, in train
[rank3]: flux_train_utils.sample_images(
[rank3]: File "/root/run/kohya/library/flux_train_utils.py", line 113, in sample_images
[rank3]: sample_image_inference(
[rank3]: File "/root/run/kohya/library/flux_train_utils.py", line 229, in sample_image_inference
[rank3]: x = denoise(flux, noise, img_ids, t5_out, txt_ids, l_pooled, timesteps=timesteps, guidance=scale, t5_attn_mask=t5_attn_mask)
[rank3]: File "/root/run/kohya/library/flux_train_utils.py", line 314, in denoise
[rank3]: pred = model(
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
[rank3]: return model_forward(*args, **kwargs)
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in __call__
[rank3]: return convert_to_fp32(self.model_forward(*args, **kwargs))
[rank3]: File "/root/miniconda/envs/kohya/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast
[rank3]: return func(*args, **kwargs)
[rank3]: File "/root/run/kohya/library/flux_models.py", line 1004, in forward
[rank3]: if img.ndim != 3 or txt.ndim != 3:
[rank3]: AttributeError: 'NoneType' object has no attribute 'ndim'

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with library/flux_train_utils.py at sample_image_inference and denoise, then inspect library/flux_models.py at forward line 1004. Reproduce sample generation during training and trace which argument becomes None before the ndim check; done means the intermittent crash no longer occurs for the affected generations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.