kohya-ss / kohya-ss/sd-scripts

RuntimeError: "eye" not implemented for 'BFloat16'

Open
#2,078 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

when I run this command, I get the error: RuntimeError: "eye" not implemented for 'BFloat16'

```bash
/home/1000/.local/bin/accelerate launch --dynamo_backend no --dynamo_mode default --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 /app/sd-scripts/sd3_train_network.py --config_file /app/outputs/config_lora/config_SD3-lora-3k-Adafactor-LR1e-4-B2-Dim128.toml 2>&1 | tee outputs/config_SD3-lora-3k-Adafactor-LR1e-4-B2-Dim128.txt
```

```
INFO Loading state dict from /app/models/clip/clip_l.safetensors sd3_utils.py:148
INFO Adding text_projection.weight to clip_l_sd sd3_utils.py:152
accelerator device: cuda
Traceback (most recent call last):
File "/app/sd-scripts/sd3_train_network.py", line 496, in
trainer.train(args)
File "/app/sd-scripts/train_network.py", line 571, in train
model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "/app/sd-scripts/sd3_train_network.py", line 112, in load_target_model
clip_l = sd3_utils.load_clip_l(
File "/app/sd-scripts/library/sd3_utils.py", line 153, in load_clip_l
clip_l_sd["text_projection.weight"] = torch.eye(768, dtype=dtype, device=device)
RuntimeError: "eye" not implemented for 'BFloat16'
Traceback (most recent call last):
File "/home/1000/.local/bin/accelerate", line 8, in
sys.exit(main())
File "/home/1000/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/1000/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/home/1000/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python3.10', '/app/sd-scripts/sd3_train_network.py', '--config_file', '/app/outputs/config_lora/config_SD3-lora-3k-Adafactor-LR1e-4-B2-Dim128.toml']' returned non-zero exit status 1.

```

file.toml
```
pretrained_model_name_or_path = "/app/models/unet/sd3.5_large.safetensors"
clip_l = "/app/models/clip/clip_l.safetensors"
clip_g = "/app/models/clip/clip_g.safetensors"
t5xxl = "/app/models/clip/t5xxl_fp16.safetensors"
train_data_dir = "/app/data/Flux大模型/train/img-13k"
logging_dir = "/app/logs"
output_dir = "/app/outputs/models"
output_name = "国风sd3.5-lora-Adafactor-LR1e-4-B2-Dim128"
train_batch_size = 2
save_every_n_epochs = 1
optimizer_type = "Adafactor"
optimizer_args = [ "relative_step=False,", "scale_parameter=False,", "warmup_init=False","weight_decay=0.01"]
learning_rate = 1e-4
text_encoder_lr = [ 1e-5, 1e-5,1e-5,]
lr_scheduler = "constant_with_warmup"
lr_scheduler_args = []
lr_scheduler_num_cycles = 1
lr_scheduler_power = 1
lr_warmup_steps = 0
bucket_no_upscale = true
network_dim = 128
network_alpha = 128
network_args = [ "train_t5xxl=True",]
epoch = 5
max_train_epochs = 5
max_train_steps = 100000000
bucket_reso_steps = 64
cache_latents = true
cache_latents_to_disk = true
cache_text_encoder_outputs = true
cache_text_encoder_outputs_to_disk = true
caption_extension = ".txt"
clip_skip = 1
discrete_flow_shift = 3.1582
dynamo_backend = "no"
enable_bucket = true
gradient_accumulation_steps = 1
guidance_scale = 1.0
huber_c = 0.1
huber_scale = 1
huber_schedule = "snr"
loss_type = "l2"
max_bucket_reso = 2048
max_data_loader_n_workers = 0
max_grad_norm = 1
max_timestep = 1000
min_bucket_reso = 256
mixed_precision = "bf16"
model_prediction_type = "raw"
network_module = "networks.lora_sd3"
noise_offset_type = "Original"
prior_loss_weight = 1
resolution = "1024,1024"
sample_prompts = "/app/outputs/sample/prompt.txt"
sample_sampler = "euler_a"
sample_every_n_epochs = 1
save_model_as = "safetensors"
save_precision = "bf16"
seed = 12345
t5xxl_max_token_length = 512
timestep_sampling = "shift"
xformers = true
sdpa = true
highvram = true
gradient_checkpointing=true

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at library/sd3_utils.py line 153, then trace how the dtype passed to load_clip_l is selected from the mixed_precision = "bf16" setting. Re-run the provided sd3_train_network.py command and verify that CLIP loading completes without the BFloat16 eye error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.