controlnet-xs-inpaint

### Describe the bug

Hi, thank you for your nice implement in conbiming controlnet-xs with diffusers. I try the code [here](https://github.com/UmerHA/diffusers/tree/cnxs-training/examples/controlnet_xs) to try the image outpaint (similiar to inpaint task), but i found a Strange thing, I don't know why, can you give me some advice?
When I train with 10,000 images, when I train 10W steps, the result is approximately:
When I train to 15W steps, the generated image looks like this:
The resulting picture in step 15W is all surrounded by some colorful dots as follow, which is strange.

### Reproduction

I use the code here : https://github.com/UmerHA/diffusers/tree/cnxs-training/examples/controlnet_xs

### Logs

_No response_

### System Info

-

### Who can help?

@sayakpaul @yiyixuxu @DN6 @patrickvonplaten

Start with the training code in examples/controlnet_xs and reproduce the reported behavior using the linked UmerHA branch. Compare the generated outputs around 100,000 and 150,000 training steps and inspect the available training configuration and logs for the source of the colorful dots. Done means identifying a reproducible cause and documenting or fixing it with evidence from the example.

Vom Indexierungsmodell aus dem Issue-Text verfasst.