SDXL drastically slows down when applying IPAdapter+ControlNets at the same time

I'm setting up a new device with RTX 4070 Ti (12GB) to run SDXL. The overall performance is within expectation, but I found it gets drastically slowed down when multiple custom nodes are applied, and is especially severed when multiple ControlNets and IPAdapter are loaded in the same workflow. The other modules (LoRa, Revision, etc.) are not having the same level of impact on speed.

i.e. For a 1200*1200 image, with IPAdapter used, when I do text to image without any ControlNet, the speed is **0.42s/it**
with only canny: 0.51s/it
with only depth: 0.51s/it
with only softedge: 0.83s/it
with canny+depth: 2.62s/it
with canny+softedge: 3.64s/it
with depth+softedge: 3.64s/it
with canny+depth+softedge: **7.96s/it**

Because of this drastic slow down, the only ways I can use it to generate images at decent speed is to either create images at small size with ControlNet applied; or create at large size with no ControlNet. The issue never happened to my previous device, assembled a year ago with RTX 3080. That one only shows a tiny decrement in speed when applying IPAdpater+ControlNets.

It seems to be a problem in resource allocation, when multiple custom progresses are competing for the same resources, like either the hardware or software environment of the new device is not configured correctly. Any one has idea on what the specific cause might be?