Sliding Sync - invites from ignored user still coming

### Description

I'm still able to receive further invites from an ignored user, on a room I was invited before and from which I've rejected the invite.
If I kill the app between ignoring the user and being reinvited, the invite is correctly ignored.
If I receive an invite from a room I've never been invited, the invite is also correctly ignored.

In any case, If I do an initial sliding sync, the invite from ignored users are correctly ignored.

### Steps to reproduce

- Get invited to a room from another user
- Reject the invite and ignore the user
- Get reinvited to the same room by the same user
- The invite comes down from the SlidingSync
- Clear cache and perform an initial sync
- The invite doesn't come down anymore

### Homeserver

matrix.org

### Synapse Version

1.126.0rc1

### Installation Method

Docker (matrixdotorg/synapse)

### Database

-

### Workers

Single process

### Platform

-

### Configuration

_No response_

### Relevant log output

```shell
-
```

### Anything else that would be useful to know?

_No response_