lllyasviel / lllyasviel/ControlNet

How to generate images using your own trained controlnet model?

Open
#465 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

Following the official training instructions, I have made the data and and trained it, also based on the test images, I found the best performance at the 22nd epoch, so I got the following from the training interrupted at the 22nd epoch:.
A file lighting_logs, I found that the updated ckpt file is stored in the lighting_logs folder, in version1 (because I trained 1 time before, the first time was version0, that time was just a trial). there are three files in version1, the first one is hparams.yaml, which has only one pair of {}. The first is hparams.yaml, which has only one pair of {}. The second one is events.out.tfevents.1689045604.autodl-container-63e511b6ae-8ebbd39d.10590.0, and the last one is the checkpoints folder, which has only one file named epoch=22- step=5335.
But I found that in the py files that call the weights for image generation, they are all either directly calling the pth file that controls controlnet:
model = create_model('. /models/cldm_v15.yaml').cpu()
model.load_state_dict(load_state_dict('. /models/control_sd15_scribble.pth', location='cuda'))
Either that or they call the pre-training ckpt weights of the sd model before calling the pth file that controls controlnet. :
model_name = 'control_v11p_sd15_scribble'
model = create_model(f'. /models/{model_name}.yaml').cpu()
model.load_state_dict(load_state_dict('. /models/v1-5-pruned.ckpt', location='cuda'), strict=False)
model.load_state_dict(load_state_dict(f'. /models/{model_name}.pth', location='cuda'), strict=False)
model = model.cuda()
Also the official yaml file they call has values:
model:
target: cldm.cldm.ControlLDM
params:
linear_start: 0.00085
linear_end: 0.0120
num_timesteps_cond: 1
log_every_t: 200
timesteps: 1000
first_stage_key: "jpg"
cond_stage_key: "txt"
control_key: "hint"
image_size: 64
channels: 4
cond_stage_trainable: false
conditioning_key: crossattn
monitor: val/loss_simple_ema
scale_factor: 0.18215
use_ema: False
only_mid_control: False

control_stage_config:
target: cldm.cldm.ControlNet
params:
image_size: 32 # unused
in_channels: 4
hint_channels: 3
model_channels: 320
attention_resolutions: [ 4, 2, 1 ]
num_res_blocks: 2
channel_mult: [ 1, 2, 4, 4 ]
num_heads: 8
use_spatial_transformer: True
transformer_depth: 1
context_dim: 768
use_checkpoint: True
legacy: False

unet_config:
target: cldm.cldm.ControlledUnetModel
params:
image_size: 32 # unused
in_channels: 4
out_channels: 4
model_channels: 320
attention_resolutions: [ 4, 2, 1 ]
num_res_blocks: 2
channel_mult: [ 1, 2, 4, 4 ]
num_heads: 8
use_spatial_transformer: True
transformer_depth: 1
context_dim: 768
use_checkpoint: True
legacy: False

first_stage_config:
target: ldm.models.autoencoder.AutoencoderKL
params:
embed_dim: 4
monitor: val/rec_loss
ddconfig:
double_z: true
z_channels: 4
resolution: 256
in_channels: 3
out_ch: 3
ch: 128
ch_mult:
- 1
- 2
- 4
- 4
num_res_blocks: 2
attn_resolutions: []
dropout: 0.0
lossconfig:
target: torch.nn.Identity

cond_stage_config:
target: ldm.modules.encoders.modules.FrozenCLIPEmbedder
May I ask what is going on here? I have considered the effect of the following
1 I didn't set the epoch number, because the official training tutorial didn't teach me how to set it, so I was stopping directly at round 22, which may have caused some loss to the model and thus nothing in the yaml
But if I need to set additional epoch number, how should I set it?
2 I set SD lock to true during training, which means that I might modify the weights of the SD model a bit as well, so the ckpt I get should be a direct replacement for the official one? And where should I get it from when PPT? Should I convert ckpt to pth?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the official training instructions and the generation Python files referenced in the report. Compare the checkpoint at lighting_logs/version1/checkpoints/epoch=22-step=5335 with the loading paths for ./models/*.yaml, *.pth, and v1-5-pruned.ckpt. Done means documenting the supported checkpoint format and reproducible steps for generating images with the trained ControlNet model.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.