VideoRectangles from predictions only appear if they all exist in the frame

**Describe the bug**
I am trying to label some VideoRectangles predictions but the tool is not displaying them until they all have a keyframe together. When trying to display many different tracks, it's then enough for one of them to not be displayed for all the others to disappear too.
I believe this is a bug, because this doesn't reproduce if I create and display the VideoRectangles manually.

**To Reproduce**
Steps to reproduce the behaviour:
1. Create Video tasks with the template pasted below
2. Upload json task containing video rectangle sequences (in my case
[task_501 (10).json](https://github.com/HumanSignal/label-studio/files/14313326/task_501.10.json))
3. Open task in frontend
4. Click on a frame where not all videoRectangles exist. See that none of the others appear.
5. Click on a frame where all of them exist. See that now, they exist.
6. You can also click on a frame where one of the VideoRectangles doesn't appear yet, and delete it. This would make the others appear..

**Expected behavior**
The VideoRectangles should all be displayed, no matter what others are displayed.

**Screenshots**
I have attached a couple of recordings, one showing the successful manual case, and the other showing the failure case, which is the one importing the predictions.

https://github.com/HumanSignal/label-studio/assets/11409469/e6df253a-7921-440d-b9ba-d4eb66a3e7a9

https://github.com/HumanSignal/label-studio/assets/11409469/d2f5d927-db59-4d7b-8472-83e050db0b70

**Environment (please complete the following information):**
- OS: MacOS but in a docker container, latest image
- Label Studio Version 1.11.0

**Additional context**
Task template:
```

```