pipe.scheduler.sigmas change after forward call

### Describe the bug

I noticed this issue when trying to manually modify the sigmas in my pipeline scheduler. Despite changing the values in this tensor, the sigmas tensor shrinks to the number of inference steps after one inference instance. Subsequent calls no longer change the values of the sigmas.

### Reproduction

pipe.scheduler.sigmas = pipe.scheduler.sigmas * 4 # manually modifying sigmas
_ = pipe(..., num_inference_steps = N)
print(pipe.scheduler.sigmas) # len is N + 1, values do not reflect the manual modifications and no longer change in subsequent calls

### Logs

_No response_

### System Info

diffusers==0.27.2

### Who can help?

_No response_