lllyasviel / lllyasviel/ControlNet

OpenPose in batch mode only generates for the first combination when used with X/Y/Z plot

Open
#404 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

When using OpenPose in batch mode with XYZ plot, it only generates "n" images (where n is the number of poses in the openpose batch folder) based on the first combination in XYZ plot.

Using a single pose is fine with XYZ Prompt but it seems like batch mode knows it had X photos in the folder, and stops generation after X images.

Logs for my test generation:

  • XYZ prompt had 3x3 (3 characters, and 3 seeds)
  • 7 poses in the openpose folder
  • 30 steps sampler +25 highresfix per image
  • ControlNet v1.1.157
  • control_v11p_sd15_openpose [cab727d4]

X/Y/Z plot will create 9 images on 1 3x3 grid. (Total steps to process: 495) controlnet batch mode Loading model from cache: control_v11p_sd15_openpose [cab727d4] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.83it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.69s/it] Loading model from cache: control_v11p_sd15_openpose [cab727d4] | 55/3465 [00:51<1:36:03, 1.69s/it] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.90it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.69s/it] Loading model from cache: control_v11p_sd15_openpose [cab727d4] | 110/3465 [01:43<1:34:35, 1.69s/it] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.96it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.70s/it] Loading model from cache: control_v11p_sd15_openpose [cab727d4] | 165/3465 [02:35<1:33:16, 1.70s/it] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.94it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.70s/it] Loading model from cache: control_v11p_sd15_openpose [cab727d4] | 220/3465 [03:28<1:31:43, 1.70s/it] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.93it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.70s/it] Loading model from cache: control_v11p_sd15_openpose [cab727d4] | 275/3465 [04:20<1:30:18, 1.70s/it] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.94it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.70s/it] Loading model from cache: control_v11p_sd15_openpose [cab727d4] | 330/3465 [05:12<1:28:47, 1.70s/it] Loading preprocessor: none preprocessor resolution = 512 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:07<00:00, 3.92it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:42<00:00, 1.70s/it] Total progress: 11%|███████ | 385/3465 [06:07<49:00, 1.05it/s] Total progress: 11%|██████▊ | 385/3465 [06:07<1:27:15, 1.70s/it]

The logs then stops and the "generate" button is back active in A1111.

At the start of generation, SD knows XYZ will create 9 images.
SD also know it should create 63 (9 images x 7 poses) images altogether based on total steps of 3465 (63 x 55 steps per image).
However after generating 7 images, it stops and the "generate" button is active again on A1111.

It looks like controlnet / openpose is only picking the first combination in the XYZ plot and then generating the images for the number of poses in the folder, and then terminates. I have confirmed the first combination is used by setting 987654321, 888888888, 123456789 as the seeds in XYZ plot and 7 resulting images were all generated with 987654321 as the seed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported setup with XYZ Plot using 3 characters, 3 seeds, and 7 OpenPose batch images, then compare the generated count and seed values with the expected 63 images. Trace the XYZ Plot and OpenPose batch interaction from the relevant ControlNet integration entry points; done means every XYZ combination is generated for each pose rather than only the first combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.