[Feature Request]:

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What would your feature do ?

The future would not load the Controlnet model, if the weight is 0.
Because I am developing the extension Deforumation that uses Deforum, I would like if the Controlnet model did not load if a user uses a weight schedule value of 0... A value of zero (to my knowledge means that the controlnet module doesn't influence the img2img render.

A simple solution that I tried is to just put these line in the function "create_cnu_dict" in "deforum_controlnet.py" (currently after row 302 (cnu['weight'] = getattr(CnSchKeys, f"cn_{model_num}_weight_schedule_series")[frame_idx]):
if cnu['weight'] == 0:
cnu['enabled'] = False

This will prevent the module to be loaded at all if the weight is 0, and also saves some time, as to not load all enabled controlnet modules. If I'm using 5 controlnet modules that I want to switch on/off realtime this really saves time.

I have tried it and it seems to work, well now with the latest fix, that you did and which closes the StableDiffusionProcessingTxt2Img pipeline.

Well met
//Rakila

### Proposed workflow

Add the two line:
if cnu['weight'] == 0:
cnu['enabled'] = False

### Additional information

_No response_

### Are you going to help adding it?

I'll let you decide upon this.