TypeError: Destination stream closed

### Steps to reproduce

1. Create meeting with `perParticipantE2EE=true`
2. Join meeting with participant A
3. Join same meeting with participant B
4. Try to reload and join again with participant B till you see the error in participant A console that destination stream closed

I saw that sometimes I'm not getting remote participant keys
How I turned on per participant e2ee:
`https://call.element.io/room/#/test?roomId=roomId&perParticipantE2EE=true`

Also I see these errors:

### Outcome

#### What did you expect?
It should not break audio/video

#### What happened instead?
Participant B cannot see/hear participant A

### Operating system

ubuntu

### Browser information

Chrome Version 115.0.5790.170 (Official Build) (64-bit)

### URL for webapp

https://call.element.io/