lllyasviel / lllyasviel/ControlNet

Clarification of some details from article

Open
#582 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

Hello. I am trying to train and infer a model according to your article. However, some things are not clear from article:
1) "In the training process, we randomly replace 50% text prompts ct with empty strings". Ok, you replace 50 % of the input text to ControlNet with empty strings, but do you so also for base model (for same elements in the batch)?
2) In section "Classifier-free guidance resolution weighting" you state that you are reweighting ControlNet residuals by wi = (64 / {resolution of residual}). Am I right, that lowest resolution residual (middle block) gets a multiplier of 8? In my case it completely breaks the generation (diffusers case of using np.linspace(-1, 0, 13) works slightly better).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the article's training-process paragraph and the section titled "Classifier-free guidance resolution weighting." Verify how the 50% empty-prompt replacement applies to the base model and ControlNet, and whether the stated residual weighting matches the described resolutions. Done means the article clearly answers both questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.