lllyasviel / lllyasviel/ControlNet
Is there a way to control controlnet fit strength with input.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 3k
- PR merge metrics
- No merged PRs in 30d
Description
I've been wondering if controlnet technically supports the ability to attenuate fit strength for a controlnet based on the region of the reference. The drawtip utility seems to imply this though that is almost non-functional when I've tried to use it.
Picture for example you had canny edge but the input image also had an alpha channel which could be 0 to max and that said "which part" of the image you want to fit how strong. This would be particular useful for controlnet using delocalized things like the reference_only mode (here an alpha mask would select which parts of the reference image to use throughout the entire image). In the WebUI verison the draw tooltip which I think is a masking feature seems to do nothing except ruin a picture, it seems completely non-functional outside of creating weird effects if you even leave one spot on your input. Being able to apply an alpha to a controlnet strength is particularly useful in a scripting pipeline, so is the underlying architecture amenable to this, is it technically already supported in all the controlnet models but just unexplored from a UI perspective, or what exactly prevents this from being regularly applied?
NOTE: I have seen multi-controlnet and latent couple used together, though in this case I'm more explicitly talking about the ability for controlnet to control the control source rather than the resulting image and how to interface with it. This seperation of concern provides more fine grained control over both in many cases since you can leverage a controlnet the best ways (if its localized like an openpose or canny then the effect on the constructed image is localized, if its delocalized then the effects are potentially image-wide but only with the parts you care about)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the drawtip utility and the WebUI masking behavior described in the issue, then trace how ControlNet receives control sources and strength values. Done should establish whether region-dependent or alpha-weighted control strength is supported and, if not, define the required interface and implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100