lllyasviel / lllyasviel/ControlNet
Clarification of some details from article
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 3k
- PR merge metrics
- No merged PRs in 30d
Description
Hello. I am trying to train and infer a model according to your article. However, some things are not clear from article:
1) "In the training process, we randomly replace 50% text prompts ct with empty strings". Ok, you replace 50 % of the input text to ControlNet with empty strings, but do you so also for base model (for same elements in the batch)?
2) In section "Classifier-free guidance resolution weighting" you state that you are reweighting ControlNet residuals by wi = (64 / {resolution of residual}). Am I right, that lowest resolution residual (middle block) gets a multiplier of 8? In my case it completely breaks the generation (diffusers case of using np.linspace(-1, 0, 13) works slightly better).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the article's training-process paragraph and the section titled "Classifier-free guidance resolution weighting." Verify how the 50% empty-prompt replacement applies to the base model and ControlNet, and whether the stated residual weighting matches the described resolutions. Done means the article clearly answers both questions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100