pipeline fail to move to "cuda" if one of the component is PeftModel

When I was doing infer, I loaded the pre-trained weights using the following method:
```
self.transformer = PeftModel.from_pretrained(
self.base_transformer,
lora_model_path
)
```
and loaded the pipeline in the following way:
```
pipe = FluxControlNetPipeline(transformer=transformer, ......)
```
However, I encountered an error when running it:
**ValueError**: It seems like you have activated sequential model offloading by calling `enable_sequential_cpu_offload`, but are now attempting to move the pipeline to GPU. This is not compatible with offloading. Please, move your pipeline `.to('cpu')` or consider removing the move altogether if you use sequential offloading.

Strangely, I have never manually set sequential model offloading. And I found that if I didn't call
```
self.transformer = PeftModel.from_pretrained(
self.base_transformer,
lora_model_path
)
```
Instead, directly set `self.transformer = self.base_transformer`, and the subsequent code remains unchanged, the above error will not occur.
If there's someone who has encountered the same issue?

### System Info

diffusers==0.33.0.dev0
peft==0.14.0

@yiyixuxu @sayakpaul @DN6

Start at the FluxControlNetPipeline entry point and trace how enable_sequential_cpu_offload and pipeline device moves are detected when the transformer is a PeftModel. Reproduce the reported snippets with diffusers 0.33.0.dev0 and peft 0.14.0, then verify that loading through PeftModel does not incorrectly block moving the pipeline to CUDA.

Escrito por el modelo de indexación a partir del texto del issue.