controlnet-xs-inpaint

### Describe the bug

Hi, thank you for your nice implement in conbiming controlnet-xs with diffusers. I try the code [here](https://github.com/UmerHA/diffusers/tree/cnxs-training/examples/controlnet_xs) to try the image outpaint (similiar to inpaint task), but i found a Strange thing, I don't know why, can you give me some advice?
When I train with 10,000 images, when I train 10W steps, the result is approximately:
When I train to 15W steps, the generated image looks like this:
The resulting picture in step 15W is all surrounded by some colorful dots as follow, which is strange.

### Reproduction

I use the code here : https://github.com/UmerHA/diffusers/tree/cnxs-training/examples/controlnet_xs

### Logs

_No response_

### System Info

-

### Who can help?

@sayakpaul @yiyixuxu @DN6 @patrickvonplaten