Clicking on permalink in search results can cause the timeline to repeatedly jump and never converge on a result

### Steps to reproduce

1. Search for a term in a busy room
2. Click on an oldish result
3. Watch the timeline try to jump to the permalink but "miss" and then spasm, repeatedly trying to paginate and jump and paginate again, and never converge; sitting there flashing on and off as it gets stuck in a loop.

### Outcome

#### What did you expect?

permalinks should work.

#### What happened instead?

permalinks caused the timeline to jump around incessantly.

### Operating system

macOS

### Application version

Nightly

### How did you install the app?

_No response_

### Homeserver

matrix.org

### Will you send logs?

No