[Community] Only half of my cpu cores are being used?

### Describe the bug

Only about half of my cpus are used.
I have 8 cores, but only 4 are used. Is there way to fix this?
^ The spikes are when diffussers is generating something

### Reproduction

My code:
```python
print("Hello, World")

import secrets
import gradio as gr
import torch
from torch import autocast
from diffusers import StableDiffusionPipeline
from PIL import Image

print("Deps loaded!")

model_id = "CompVis/stable-diffusion-v1-4"
device = "cpu"

pipe = StableDiffusionPipeline.from_pretrained(model_id, use_auth_token=True)
pipe = pipe.to(device)

print("Loaded!")

def predict(name):
print(f"Prompt: {name}")
prompt = name
with autocast("cuda"):
image = pipe(prompt, guidance_scale=7.5, width=512, height=512, num_inference_steps=20).images[0]

id = secrets.token_urlsafe(16)
image.save(f"./out/{id}.png")

return image

print("Starting...")
demo = gr.Interface(
predict,
inputs=[
gr.inputs.Textbox(label='Prompt', default='a chalk pastel drawing of a llama wearing a wizard hat')
],
outputs=gr.Image(shape=[512,512], type="pil", elem_id="output_image"),
css="#output_image{width: 512px; height: 512px}",
title="Retslav - Text To Image - Stable Diffusion",
description="Retslav Stable Diffussion",
)

demo.launch(server_port=3000)
```

### Logs

```shell
-
```

### System Info

- `diffusers` version: 0.6.0
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.13.0+cpu (False)
- Huggingface_hub version: 0.10.1
- Transformers version: 4.23.1
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

Start with the provided reproduction script, especially StableDiffusionPipeline and the .to("cpu") path, and reproduce the reported utilization on the stated Python, PyTorch, diffusers, and Linux versions. Inspect how CPU generation and thread usage are configured, then document a confirmed cause and a reproducible fix or limitation; done means generation uses the expected available cores or the behavior is clearly explained.

Scritto dal modello di indicizzazione a partire dal testo della issue.