kohya-ss / kohya-ss/sd-scripts

loss and lr not record on wandb

Open
#494 11 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

![image](https://github.com/kohya-ss/sd-scripts/assets/94830014/0b30d966-f4c9-4c72-ad01-672389b960c5)

I attempted to record the loss and learning rate of my lora learning, but only GPU information was recorded. My config.toml file contains the following settings:

log_with = "wandb"
log_tracker_name = "lora_0511"
wandb_api_key = "apikey"

pretrained_model_name_or_path = "....ckpt"
train_data_dir = "..."

shuffle_caption = true
caption_extension = ".txt"
keep_tokens = 20
resolution = "768"
vae_batch_size = 4
enable_bucket = true
output_dir = "..."
output_name = "..."
save_precision = "fp16"
save_every_n_epochs = 10

train_batch_size = 2
gradient_checkpointing = true
gradient_accumulation_steps = 64

max_token_length = 150
xformers = true
max_train_epochs = 50
persistent_data_loader_workers = true
seed = 42
mixed_precision = "bf16"
clip_skip = 2

multires_noise_iterations = 6
multires_noise_discount = 0.1

flip_aug = true
use_8bit_adam = true
lr_scheduler = "cosine_with_restarts"
lr_warmup_steps = 12
lr_scheduler_num_cycles = 10
unet_lr = 0.0004
text_encoder_lr = 0.0002
network_module = "networks.lora"
network_dim = 64
network_alpha = 32.0

https://github.com/kohya-ss/sd-scripts/pull/428
I read this page, and know it's ok to ignore "logging_dir"

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the configuration from the issue, using config.toml and the W&B settings shown, then compare the recorded GPU information with the missing loss and learning-rate values. Review the guidance in pull request #428 and confirm that a completed run records both metrics in W&B while retaining the existing GPU data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.