ControlNet and Unet are simultaneously loaded into VRAM

Im trying to use the SDXL [Canny ControlNet model](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0/tree/main) but i only have 6GB of VRAM and the ksampler will try to load the SDXL Unet 4.6GB and the ControlNet 2.5GB at the same time ending up with a cuda out of memory error.

I tried to use the WAS Suite Store and Load cache nodes. The conditioning variables seems to have the instructions about how to build the embeddings by seeing a cache file of 2.6GB instead of a vector of embeddings that I was expecting.
From my understanding of LDMs, the Unet(model variable) only needs the embeddings and the latent image and the ControlNet is a separate model that predicts the text embeddings that would better match the given prompt and the reference image. I think the embeddings remain constant through the inference of the Unet, I'm not sure if the sampler or the scheduler would ever modify the embeddings.
If the embeddings are constant then there should be a way to optimize the memory allocation by calculating the embeddings first and then the inferencing the Unet once there is space in the vram.

My question is if already is an extension that can calculate the the embeddings of the controlnet and then free the vram space for inferencing the Unet, this could be implemented either as a node taking conditioning as input and returning another conditioning that has calculated the embeddings, no longer needs the controlnet model. Or creating another apply controlnet node that directly does the computation returning a conditioning that doesnt uses less memory and wont be loaded into the VRAM when inferencing the Unet.

Note this [workflow.json](https://github.com/comfyanonymous/ComfyUI/files/12328103/workflow.json.txt) will create a 2.6GB file!