Control-lite still not working properly

There was a precision mix match that was fixed a while back, but it never actually fixed the issue with these controlnets themselves. I know a very long time ago, right around when SDXL controlnets first came out, these worked just fine, but every single controlnet output puts out a distorted or garbled output when used in a weight of 0.6 to 1. Or basically, the only way to get a non garbled output is to put the weight soo low that it doesn't really follow the controlnet input image at all anyways. I've experimented with lowering the controlnet time steps, and it doesn't really fix the issue either. I've also experimented with all the other settings like balanced/prompt is more important/controlnet is more important, in combination with different settings of all the other previous controlnet options I mentioned, and it still doesn't fix the issue.

I get these issues with ALL controllite models. Whether it's kohya or bdsqlz controlnets, whether it's normal/canny/depth/etc. I've tried different samplers, all kinds of webui settings, literally everything you can tweak, I've tried, and ALL controllite controlnets are all essentially unusable.

These examples include a weight of 1. If I go to 0.8 or 0.5, you essentially get nothing like the actual input. This is about as close to the actual input it can get. The first 2 images are the preprocessors, and the second two images are the final outputs. Each processed separately. The third image is from a softedge output, and the final image is from the depth output, each using the correct softedge/depth controlnet models to generate them with.

So basically, someone borked something at some point in webui in one of the updates from a long time ago, and its all been broken ever since. I think some update in the code for how things are handled with controlnet and control-lite, didn't consider some aspect of how it functions in conjunction with other things, and is likely the source of this issue.

Start by reproducing the issue with the described ControlNet Lite models, preprocessors, weights, samplers, and control modes, comparing the input and generated images. Done means ControlNet Lite outputs are no longer distorted or garbled at usable weights while still following the control image; the issue names no files or tests to inspect.