lllyasviel / lllyasviel/ControlNet

controlNet - convert sketch circuit to vector-engineered circuit,some question

Open
#526 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

I want to achieve what is shown in the diagram below, converting a hand-drawn circuit diagram into a vector-engineered diagram. eg.
![image](https://github.com/lllyasviel/ControlNet/assets/23551774/1ff4c25b-5f3c-4bf5-85f0-10d98fcf8cb7)

all the controlText is : {"text": "Convert this sketch circuit diagram into a standard vectorized circuit diagram created with software", "image": "images/1000_1_360015968.jpg", "conditioning_image": "conditioning_images/1000_1_360015968.jpg"}
I have prepared 50,000 similar data pairs, with a sketch circuit diagram corresponding to an identical vector image.
Then, I trained ControlNet with
```
batchsize=1(my gpu is one 3090),
lr = 2e-6,
precision=32,
accumulate_grad_batches=4,
sd_locked = False,
only_mid_control = False,
```
Roughly after 2 epochs, there wasn't much change in the test simple images. The input sketch didn't seem to have any controlling effect, and there was a tendency for overfitting. Regardless of the input sketch, the output results were random combinations of circuit diagrams.
here are some egs:
text: Convert this sketch circuit diagram into a standard vectorized circuit diagram created with software
control image:
![image](https://github.com/lllyasviel/ControlNet/assets/23551774/6025e4f0-b1cd-4dd3-992b-e2428c9d1e96)
output image:
![image](https://github.com/lllyasviel/ControlNet/assets/23551774/03965144-9bd0-4281-af99-c00fcec55ed4)

can some one tell me some advice? thanks!

__________________________________________________________________________________________
我想实现的内容就是把手绘的物理电路图,通过controlNet控制输出类似工程软件绘制的电路图,示例图如上面的所以,
我目前准备了50K的手绘草图和对应的矢量图,找人一比一绘制出来的。
控制的text全部都是固定成了一句话,“把手绘的电路图转成工程的矢量图”,

在2个epoc之后,基本上就稳定了,不管输入的是什么控制图,输出的都是随机组合的电路图,样式上还可以,但是就是没有起到控制作用,跟输入的草图完全不一样!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the paired sketch/vector dataset and the training configuration reported in the issue, then compare conditioning images with generated outputs across training steps. The work is complete only when outputs preserve the input circuit structure rather than producing unrelated circuit diagrams; no project files or tests are identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.