lllyasviel / lllyasviel/stable-diffusion-webui-forge

ControlNet and Kohya HRFix interop support

Open
#95 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Is there an existing issue for this?
  • I have searched the existing issues and checked the recent builds/commits
What would your feature do ?

我经常使用高分辨率的图生图,会使用CN的lineArt和Tile两个模型的样子。而且我发现,小分辨率的时候(1000px),SD生成的图,光影和结构更加正确、优秀,超过2000px的时候就会变得很平、很容易出错的样子,无论是结构上,还是光影。

然后我使用了内置的Kohya HRFix,它的工作机制就是先缩放分辨率迭代出正确的结构,然后再提升会原本的分辨率,而且可以控制参与的步骤的样子。在文生图(1500x1500,甚至2000x2000以上),能得到结构正确且非常优秀的画面。
我就想着试着搭配CN模型一起,进一步控制画面的样子,然后就出现了报错的样子,在Kohya HRFix起作用的阶段中,CN就会出现
warning control could not be applied torch.Size([2, 1280, 11, 8]) torch.Size([2, 1280, 32, 22])
warning control could not be applied torch.Size([2, 1280, 11, 8]) torch.Size([2, 1280, 32, 22])
warning control could not be applied torch.Size([2, 1280, 11, 8]) torch.Size([2, 1280, 32, 22])
warning control could not be applied torch.Size([2, 1280, 22, 15]) torch.Size([2, 1280, 64, 43])
warning control could not be applied torch.Size([2, 1280, 22, 15]) torch.Size([2, 1280, 64, 43])
warning control could not be applied torch.Size([2, 640, 22, 15]) torch.Size([2, 640, 64, 43])
warning control could not be applied torch.Size([2, 640, 43, 29]) torch.Size([2, 640, 128, 86])
warning control could not be applied torch.Size([2, 640, 43, 29]) torch.Size([2, 640, 128, 86])
然后并不起作用的样子。因为这个分辨率和CN设定的原本的分辨率不匹配的样子。

所以我特别希望CN也能自动匹配和Kohya HRFix的流程。

Proposed workflow
  1. 到图生图/文生图,写好参数和设置
  2. 启用Kohya HRFix Integrated,设置好参数
  3. 启用CN,设置好参数和模型
  4. CN能和HRFix一起正常运转
Additional information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the documented text-to-image or image-to-image workflow with Integrated Kohya HRFix and ControlNet, then inspect the ControlNet integration at the HRFix stage where the reported tensor-shape warnings occur. Done means ControlNet applies during HRFix without those warnings and preserves the configured conditioning.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning, python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.