Poor performance when using two or more `DirectionalLight`s with different settings

## Bevy version

0.10

## \[Optional\] Relevant system information

> 2023-03-19T14:12:24.865565Z INFO bevy_render::renderer: AdapterInfo { name: "NVIDIA GeForce GTX 970", vendor: 4318, device: 5058, device_type: DiscreteGpu, driver: "NVIDIA", driver_info: "525.85.05", backend: Vulkan }
> 2023-03-19T14:12:24.974571Z INFO bevy_diagnostic::system_information_diagnostics_plugin::internal: SystemInfo { os: "Linux 22.04 Pop!_OS", kernel: "6.2.0-76060200-generic", cpu: "AMD Ryzen 5 5600G with Radeon Graphics", core_count: "6", memory: "15.4 GiB" }

## What you did

1. Spawn a scene with objects and two directional lights with shadows enabled.
2. Either a) set one of the lights to `Visibility::Hidden` or b) set one of the lights to `shadows_enabled: false`.

Note that this doesn't work if you change *both* of the lights, or only have one light.

## What went wrong

FPS is massively degraded.

## Additional information

Encountered in https://github.com/Leafwing-Studios/Emergence/pull/542; feel free to pull and build there if you can't reproduce more simply.

Simply setting the light intensity to 0. does not cause the same problems.

When only one directional light is enabled, disabling it (or its shadows), results in a serious FPS improvement, as expected.

I used the same shadow map config for both; I haven't experimented to see if these issues are repeated if only that differs.