lllyasviel / lllyasviel/ControlNet
How to generate images using your own trained controlnet model?
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 3k
- PR merge metrics
- No merged PRs in 30d
Description
Following the official training instructions, I have made the data and and trained it, also based on the test images, I found the best performance at the 22nd epoch, so I got the following from the training interrupted at the 22nd epoch:.
A file lighting_logs, I found that the updated ckpt file is stored in the lighting_logs folder, in version1 (because I trained 1 time before, the first time was version0, that time was just a trial). there are three files in version1, the first one is hparams.yaml, which has only one pair of {}. The first is hparams.yaml, which has only one pair of {}. The second one is events.out.tfevents.1689045604.autodl-container-63e511b6ae-8ebbd39d.10590.0, and the last one is the checkpoints folder, which has only one file named epoch=22- step=5335.
But I found that in the py files that call the weights for image generation, they are all either directly calling the pth file that controls controlnet:
model = create_model('. /models/cldm_v15.yaml').cpu()
model.load_state_dict(load_state_dict('. /models/control_sd15_scribble.pth', location='cuda'))
Either that or they call the pre-training ckpt weights of the sd model before calling the pth file that controls controlnet. :
model_name = 'control_v11p_sd15_scribble'
model = create_model(f'. /models/{model_name}.yaml').cpu()
model.load_state_dict(load_state_dict('. /models/v1-5-pruned.ckpt', location='cuda'), strict=False)
model.load_state_dict(load_state_dict(f'. /models/{model_name}.pth', location='cuda'), strict=False)
model = model.cuda()
Also the official yaml file they call has values:
model:
target: cldm.cldm.ControlLDM
params:
linear_start: 0.00085
linear_end: 0.0120
num_timesteps_cond: 1
log_every_t: 200
timesteps: 1000
first_stage_key: "jpg"
cond_stage_key: "txt"
control_key: "hint"
image_size: 64
channels: 4
cond_stage_trainable: false
conditioning_key: crossattn
monitor: val/loss_simple_ema
scale_factor: 0.18215
use_ema: False
only_mid_control: False
control_stage_config:
target: cldm.cldm.ControlNet
params:
image_size: 32 # unused
in_channels: 4
hint_channels: 3
model_channels: 320
attention_resolutions: [ 4, 2, 1 ]
num_res_blocks: 2
channel_mult: [ 1, 2, 4, 4 ]
num_heads: 8
use_spatial_transformer: True
transformer_depth: 1
context_dim: 768
use_checkpoint: True
legacy: False
unet_config:
target: cldm.cldm.ControlledUnetModel
params:
image_size: 32 # unused
in_channels: 4
out_channels: 4
model_channels: 320
attention_resolutions: [ 4, 2, 1 ]
num_res_blocks: 2
channel_mult: [ 1, 2, 4, 4 ]
num_heads: 8
use_spatial_transformer: True
transformer_depth: 1
context_dim: 768
use_checkpoint: True
legacy: False
first_stage_config:
target: ldm.models.autoencoder.AutoencoderKL
params:
embed_dim: 4
monitor: val/rec_loss
ddconfig:
double_z: true
z_channels: 4
resolution: 256
in_channels: 3
out_ch: 3
ch: 128
ch_mult:
- 1
- 2
- 4
- 4
num_res_blocks: 2
attn_resolutions: []
dropout: 0.0
lossconfig:
target: torch.nn.Identity
cond_stage_config:
target: ldm.modules.encoders.modules.FrozenCLIPEmbedder
May I ask what is going on here? I have considered the effect of the following
1 I didn't set the epoch number, because the official training tutorial didn't teach me how to set it, so I was stopping directly at round 22, which may have caused some loss to the model and thus nothing in the yaml
But if I need to set additional epoch number, how should I set it?
2 I set SD lock to true during training, which means that I might modify the weights of the SD model a bit as well, so the ckpt I get should be a direct replacement for the official one? And where should I get it from when PPT? Should I convert ckpt to pth?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the official training instructions and the generation Python files referenced in the report. Compare the checkpoint at lighting_logs/version1/checkpoints/epoch=22-step=5335 with the loading paths for ./models/*.yaml, *.pth, and v1-5-pruned.ckpt. Done means documenting the supported checkpoint format and reproducible steps for generating images with the trained ControlNet model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100