Removals may cause duplicate entries (at least on message reordering), making undoing them harder

Fast removals may cause duplicate entries, this can make it hard to get a good list of who was removed since there will be many entries shown:

- Operating System (Linux/Mac/Windows/iOS/Android): Linux
- Core Version: v2.22.0
- Client Version:2.22.0 (git: flathub)

## Expected behavior

No duplicate entries when people are removed.

## Actual behavior

Duplicate entries!

### Steps to reproduce the problem

1. Don't ask me what happened on the remover's end to trigger this, I have no idea.

### Screenshots

see above.

### Logs