Increasing RAM usage with enable_model_cpu_offload

### Describe the bug

When using enable_model_cpu_offload on StableDiffusionXLPipeline, each consecutive __call__ takes more and more RAM. Also, after deleting pipe not all memory is freed

### Reproduction

```python
import gc
import torch
from diffusers import StableDiffusionXLPipeline
import psutil

def print_memory_usage(step):
print(f"{step} - Memory usage: {psutil.virtual_memory().used / (1024 ** 3):.2f} GB")

def clear_memory():
torch.cuda.empty_cache()
gc.collect()

def inference():
print_memory_usage("Before loading pipeline")

# Load the pipeline
pipe = StableDiffusionXLPipeline.from_pretrained(
"4spaces/RealVisXL_V4.0",
torch_dtype=torch.float16,
variant="fp16",
)
pipe.enable_model_cpu_offload()

print_memory_usage("After loading pipeline")

# Move the model to CPU
pipe.to("cpu")

print_memory_usage("After moving model to CPU")

# Generate an image and clear memory
for i in range(3):
_ = pipe("horse")
print_memory_usage(f"After generating {i + 1}")
clear_memory()

# Delete the pipeline
del pipe
clear_memory()

print_memory_usage("After deleting pipeline")

inference()
clear_memory()
print_memory_usage("After inference")
```

### Logs

```shell
Before loading pipeline - Memory usage: 0.71 GB
Loading pipeline components...: 100%|██████████████████████████████████| 7/7 [00:03<00:00, 1.98it/s]
After loading pipeline - Memory usage: 0.80 GB
After moving model to CPU - Memory usage: 0.80 GB
100%|████████████████████████████████████████████████████████████████| 50/50 [00:39<00:00, 1.27it/s]
After generating 1 - Memory usage: 7.87 GB
100%|████████████████████████████████████████████████████████████████| 50/50 [00:37<00:00, 1.34it/s]
After generating 2 - Memory usage: 9.62 GB
100%|████████████████████████████████████████████████████████████████| 50/50 [00:37<00:00, 1.35it/s]
After generating 3 - Memory usage: 9.87 GB
After deleting pipeline - Memory usage: 9.37 GB
After inference - Memory usage: 7.35 GB
```

### System Info

- 🤗 Diffusers version: 0.28.0.dev0
- Platform: Clear Linux OS - Linux-6.1.71-427.aws-x86_64-with-glibc2.38
- Running on a notebook?: No
- Running on Google Colab?: No
- Python version: 3.11.0
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.23.0
- Transformers version: 4.40.2
- Accelerate version: 0.30.1
- PEFT version: 0.11.0
- Bitsandbytes version: 0.42.0
- Safetensors version: 0.4.3
- xFormers version: 0.0.23
- Accelerator: Tesla T4, 15360 MiB VRAM
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@yiyixuxu @sayakpaul