Issues reconnecting

I'm trying to create a video chat room where people can join and leave.

First, user signs up and joins the socket room. He is assigned a unique peerId which is saved on the socket alongside the socket.id. When he leaves and joins again - he is assigned a new peerId (I did this in order to avoid handling using the same peerIds)

When a new user joins:
```
socket.on("video_users", (data: any) => {
if (!this.state.id) {
toast.error("Error joining the call, try to refresh");
return;
}

data.peers.map((peer: any) => {
const id = peer.id;
if (!this.peers[id] && id !== this.state.peerId) {
this.peers[id] = this.generatePeer(
peer.id,
peer.socketId,
initiator,
data.iceServers //comes from server which creates a unique iceServer string
);
}
});
});
```

He then signals:

```
generatePeer(
peerId: string,
socketId: string,
initiator: boolean = false,
iceServers: any = null
) {
const peer = new Peer({
initiator,
trickle: false,
config: {
iceServers,
},
stream: this.state.stream,
});

peer.on("signal", (data: any) => {
if (this.state.peerId !== peerId) {
socket.emit("video_signal", {
signal: data,
to: socketId,
name: this.state.name,
host: this.state.host,
iceServers,
});
}
});

peer.on("close", () => {
delete this.peers[peerId];
})
....
}
```

The signal is catched by:

```
socket.on("video_new_signal", async (data: any) => {
const socketId = data.socketId;
const peerId = data.peerId;
if (peerId !== this.state.peerId) {
if (!this.peers[peerId]) {

this.peers[peerId] = this.generatePeer(
peerId,
socketId,
false,
data.iceServers //passed by the server
);
}
if (!this.peers[peerId].destroyed) this.peers[peerId].signal(data.signal);
}
});
```

Lastly, when a user leaves the call this function is being called:
```
endCall = () => {
Object.keys(this.peers).map((id: string) => {
this.peers[id].destroy();
});
};
```

Everything works fine on first join.

Now, the problem is I get error saying "Cannot set remote answer in state stable". Why isn't it properly destroyed (aka still stable)? I'm a bit confused of what I am doing wrong here.