Clients incorrectly denied permission to view some events

This issue has been migrated from [#10369](https://github.com/matrix-org/synapse/issues/10369).

---

### Description

Loading the start of the time timeline in some rooms fails.

### Steps to reproduce

- find a room that is marked unread in element-web
- try to access the first message
- You get the "Failed to load timeline position: tried to load a specific point in this room's timeline, but you do not have permission to view the message in question." error message

Note: this does not apply to all rooms, but in our instance to a lot of rooms.

Expectation: old events can always be loaded.

Further investigating shows that a request to

https://ungleich.matrix.ungleich.cloud/_matrix/client/r0/rooms/!qlfxvXwtSjmInWnicT%3Aungleich.ch/context/%24PsaDH5PnSV5u4lx3-06R2R9-vbDWUU8ErR2hOntC5e4?filter=%7B%22lazy_load_members%22%3Atrue%7D

fails with:

`{"errcode":"M_FORBIDDEN","error":"You don't have permission to access that event."}`

I first thought that might be related to us running https://github.com/matrix-org/rust-synapse-compress-state/issues/27 , however we only compressed rooms with 10k+ events and many rooms that are much smaller are affected.

The users trying to access the events are often even an admin in the room, so a permission issue is unlikely.

### Version information

- **Homeserver**: ungleich.matrix.ungleich.cloud

If not matrix.org:

- **Version**: {"server_version":"1.37.1","python_version":"3.7.3"}

- **Install method**: package manager

- **Platform**: Debian 10, no containers