self-trained ControlNet is slower than standard

Nobody has claimed this yet.

I train myself ControlNet according to [tutorial_train.py](https://github.com/lllyasviel/ControlNet/blob/main/tutorial_train.py).

After training, I got ```my_cn.ckpt```, size about 8G.

```my_cn.ckpt``` can load, run and get expected results by [gradio_scribble2image.py](https://github.com/lllyasviel/ControlNet/blob/main/gradio_scribble2image.py) , just update:
```python
model.load_state_dict(load_state_dict('./models/my_cn.ckpt', location='cuda'))
```

However, during inference, I found ```my_cn``` is ***several times slower*** than yours [```huggingface```](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_scribble.pth).

I print ```state_dict``` in ```my_cn.ckpt``` and ```control_sd15_scribble.pth```, both are ```torch.float32```.

I test ControlNet alone, code as follow:
```python
from share import *
import cv2
import torch
from cldm.model import create_model, load_state_dict
from cldm.ddim_hacked import DDIMSampler
from tqdm import tqdm
from torchinfo import summary

model = create_model('./models/cldm_v15.yaml').cpu()

# 317 ms
# 1445.12M-params
sketch_ckpt_path = './models/control_sd15_scribble.pth'

# 1545 ms
# 1445.12M-params
# sketch_ckpt_path = './models/my_cn.ckpt'

model.load_state_dict(load_state_dict(sketch_ckpt_path, location='cuda'))

model = model.cuda()
control_net = model.control_model

x, hint, timesteps, context = torch.rand((1,4,64,64)).to('cuda'), torch.rand((1,3,512,512)).to('cuda'), torch.rand((1)).to('cuda'), torch.rand((1,77,768)).to('cuda')

# print model information: https://github.com/TylerYep/torchinfo
summary(control_net, input_data=[x, hint, timesteps, context])

epoch = 50
e_sum = 0.00
for i in tqdm(range(0, epoch)):
begin = cv2.getTickCount()
control_net(x, hint, timesteps, context)
end = cv2.getTickCount()
# to ms
e_sum += (end - begin) / cv2.getTickFrequency() * 1000.0
print(e_sum / epoch)
print("Done!")
```

I think I must have missed some details, looking forward to your suggestions.

Start by reproducing the timing comparison in the issue with control_sd15_scribble.pth and my_cn.ckpt, then inspect tutorial_train.py, gradio_scribble2image.py, and the model-loading path used by cldm.model. Done means identifying the cause of the checkpoint-dependent slowdown and documenting a verified fix or explanation.

Written by the indexing model from the issue text.