Simple straight collisions cause loss of momentum, eventually stopping motion entirely

So I've hit some funkyness with the conservation of energy during simulations.

Windows 10
- Window focused
Conserves all energy (even with rounding errors)
- Minimizing frequently
Conserves rather well (extreemly rare are the cases when it dips and by very little, dip != floating errors)

Mac OS (latest ventura)
- Window focused
Dips randomly
- Minimizing window / swapping active workspace
Dips often and way harder to the point at which it stops all movement eventually in ~5-10 min

All forces (restitution, dampening, gravity, friction etc) are configured to keep the system going forever but in reality the ball eventually stops completely.
I'm aware that the window in bevy may be throttled but by default it runs as fast as it can.

Tried using fixed timestep, increasing frequency etc. nothing really worked.
Saw [integration parameters](https://rapier.rs/docs/user_guides/rust/integration_parameters/) but from reading about them not sure what I could try.

What I noticed is that swapping workspaces, minimizing windows exhacerbates the loss but it happens regardles, extreemly noticeable on mac.

```toml
[dependencies]
bevy = "0.10.0"
bevy_rapier2d = "0.21.0"

[profile.dev]
opt-level = 1
[profile.dev.package."*"]
opt-level = 3
```
[main.rs](https://github.com/dimforge/bevy_rapier/files/11063591/main.rs.txt)

[recording](https://user-images.githubusercontent.com/4136467/227562575-75684836-ebfd-4fbb-9b11-4228de5d58e1.mov)
[log-release-workspace-switch.txt](https://github.com/dimforge/bevy_rapier/files/11063707/log-release-workspace-switch.txt)
[log-release-same-workspace-but-minimizing.txt](https://github.com/dimforge/bevy_rapier/files/11063708/log-release-same-workspace-but-minimizing.txt)