Frame rate permanently drops overtime

## Bevy version

0.12.1

## \[Optional\] Relevant system information

- Windows 10
- cargo 1.76.0-nightly (2c03e0e2d 2023-11-16)
- tested on two machines : i5, 16gb mem, gtx 970 and i5, 8gb mem, gtx 670

## What you did

Created a test program displaying the framerate, and left it running for 8+ hours.

## What went wrong

Was expecting framerate to stay at 60 fps. But framerate would decrease over time, down to about 15fps and never recover.

## Additional information

Framerate decreases faster when:

* more going on (eg lots of entities/components, rendering)
* when running another program that has high cpu/gpu usage at the same time as the bevy app

With a barebones bevy app, it can take up to a day or possibly longer for the framerate to decrease.

In my bevy app, the menus are noticeably laggy to interact with after the framerate has dropped to 50-45 fps.

### Test code

`Cargo.toml` :

```toml
[package]
name = "hello"
version = "0.1.0"
edition = "2021"

[dependencies]
bevy = "0.12.1"
```

`rust-toolchain.toml` :

```toml
[toolchain]
channel = "nightly"
```

`main.rs` :

```rust

use bevy::diagnostic::{FrameTimeDiagnosticsPlugin, DiagnosticsStore};
use bevy::prelude::*;

fn main() {
let mut app = App::new();

app
.add_plugins(( bevy::prelude::DefaultPlugins, FrameTimeDiagnosticsPlugin::default(), ))
.add_systems(Update, ( show_fps_in_title, )) ;

app.run();
}

fn show_fps_in_title( mut windows: Query<&mut Window>, diagnostics: Res, ) {
let Ok(mut window) = windows.get_single_mut() else { return; };

let (fps,avg,smoothed) = diagnostics.get(FrameTimeDiagnosticsPlugin::FPS)
.map(|x|(x.value().unwrap_or_default(),x.average().unwrap_or_default(),x.smoothed().unwrap_or_default()))
.unwrap_or_default();

window.title = format!("{fps:.0} {avg:.0} {smoothed:.0}");
}
```