Invites are not always included in sync

This issue has been migrated from [#7406](https://github.com/matrix-org/synapse/issues/7406).

---

### Description

On fast changing membership states the sync will not reliably include rooms a user was invited to.
It might be related to issues
https://github.com/matrix-org/synapse/issues/4533 and
https://github.com/matrix-org/synapse/issues/4422

### Reproduction

This is not easily reproducable, an attempt should look as follows:

- Roomadmin a invites user b into rooms r_a and r_b
- a kicks b from r_a before he accepts the invite
- a changes room_state of r_b multiple times
- b retrieves current rooms via (initial)sync, r_b doesn't show

### Version information

We used the synapse version v1.12.0.
Installed per git clone
And is operating in a debian stretch docker container.

### Hotfix

A possibility to fix it temporarly is to restart the server or invalidate the caches.

We debugged this issue for quite some time and are using a hotfix, that stops caching from
`get_forgotten_rooms_for_user` in roommember.py by removing following line:
https://github.com/matrix-org/synapse/blob/9c1b83b0078aa9cc1bb902e14d3f7302625ba099/synapse/storage/data_stores/main/roommember.py#L797