custom_pipeline not being cached

### Describe the bug

Components of custom pipelines (defined via custom_pipeline parameter) do not seem to be cached.
I have a multistep system where I'm loading all my components and then caching them explicitly using transformers.utils.move_cache.
I have an sdxl pipeline with `custom_pipeline="lpw_stable_diffusion_xl"`. The normal pipeline components seem to be cached correctly but not the components of the custom_pipeline.

I'm restoring the cached components by using the Pipeline with `local_files_only=True`, however diffusers is always trying to reach the internet for the custom_pipeline components, which it cannot, since the env where the restoration happens, doesn't have a connection.

### Reproduction

```python
StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16,
custom_pipeline="lpw_stable_diffusion_xl"
)
```

### Logs

_No response_

### System Info

latest diffusers version from main branch

### Who can help?

@yiyixuxu @sayakpaul @DN6