Unloading multiple loras: norms do not return to their original values

When unloading from multiple loras on flux pipeline, I believe that the norm layers are not restored [here](https://github.com/huggingface/diffusers/blob/464374fb87610c53b2cf81e08d3df628fada3ce4/src/diffusers/loaders/lora_pipeline.py#L1575).

Shouldn't we have:

```python
if len(transformer_norm_state_dict) > 0:
original_norm_layers_state_dict = self._load_norm_into_transformer(
transformer_norm_state_dict,
transformer=transformer,
discard_original_layers=False,
)
if not hasattr(transformer, "_transformer_norm_layers"):
transformer._transformer_norm_layers = original_norm_layers_state_dict
```