Speed up Pods installation

Is there any way to speed up / skip Pods installation once all the Pods have been already pre-build and are available?

```
-----------------------------------------
🚀 Prebuild frameworks
-----------------------------------------
...
Validate prebuilt cache
Cache validation: hit (81) [...]
Prebuilding...
Installing [Podname]...
Installing [Podname]...
...
Prebuild frameworks (total 0): []
No changes in prebuild
```

I'm running `cocoapods-binary-cache` on the CI and the step above is taking over 50% of the time. It is actually taking as long as if I was gonna build all dependencies without using the plugin, so I'm not gaining any time saved.

To provide more details I'm using Github Actions and even though the task above takes 20 seconds to finish locally on my computer it is taking 7 minutes and 30 seconds on the CI. Compiling and running the rest of the app is then taking 6 minutes.

*I've tried using Ruby `3.0.0` & `3.1.0`

Edit: I've noticed there are properties like `prebuild_job`, `prebuild_all_pods`, but setting them up seems to have no effect.