Unable to leave a certrain room (KeyError).

### Description

My account on the Synapse instance can't leave `#offtopic:archlinux.org`. Element said MatrixError: [500] Internal server error.

### Steps to reproduce

- Trying to leave #offtopic:archlinux.org on any client
- Receive the error

### Homeserver

moechat.kimiblock.top

### Synapse Version

1.98.0rc1

### Installation Method

Other (please mention below)

### Database

Running a single instance of PostgreSQL

### Workers

Single process

### Platform

- OS: Arch Linux
- Synapse [Self Packaged]: Using PKGBUILD from [here](https://gitlab.archlinux.org/archlinux/packaging/packages/matrix-synapse) and replaced `pkgver` and `_tag` for 1.98 rc1
- PostgreSQL: 16.1-1
- Python: 3.11.6-1
- Architecture: x86_64
- Processor: Ryzen 7 5700U

### Configuration

mscs:

```
experimental_features:
msc3773_enabled: true
msc3391_enabled: true
msc3890_enabled: true
msc3266_enabled: true
msc4028_enabled: true
msc3026_enabled: true
```

Sliding Sync Proxy do exist, message retention set to:

```
retention:
enabled: true
default_policy:
min_lifetime: 60d
max_lifetime: 90d
allowed_lifetime_min: 90d
allowed_lifetime_max: 180d
purge_jobs:
- longest_max_lifetime: 3d
interval: 12h
- shortest_max_lifetime: 3d
interval: 1d
```

Connected to Mautrix Telegram bridge, but that shouldn't affect this.

### Relevant log output

```shell
Dec 18 20:23:06 serverOS-kimiblock.top synapse[566873]: 2023-12-18 20:23:06,952 - synapse.access.http.60000 - 465 - INFO - PUT-2638222 - 127.0.0.1 - 60000 - {flytothe.top} Processed request: 0.026sec/0.002sec (0.004sec, 0.001sec) (0.001sec/0.009sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1702902184675-203 HTTP/1.1" "Dendrite/0.13.5+719858d" [0 dbevts]
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: 2023-12-18 20:23:07,057 - synapse.http.server - 140 - ERROR - POST-2638226 - Failed handle request via 'RoomMembershipRestServlet':
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: Traceback (most recent call last):
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/http/server.py", line 326, in _async_render_wrapper
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: callback_return = await self._async_render(request)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/http/server.py", line 538, in _async_render
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: callback_return = await raw_callback_return
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1072, in on_POST
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: return await self._do(request, requester, room_id, membership_action, None)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1045, in _do
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: await self.room_member_handler.update_membership(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 645, in update_membership
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: result = await self.update_membership_locked(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 1104, in update_membership_locked
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: return await self._local_membership_update(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 482, in _local_membership_update
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: await self.event_creation_handler.handle_new_client_event(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/util/metrics.py", line 113, in measured_func
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: r = await func(self, *args, **kwargs)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/handlers/message.py", line 1493, in handle_new_client_event
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: result, _ = await make_deferred_yieldable(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: result = context.run(gen.send, result)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/handlers/message.py", line 1571, in _persist_events
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: event = await self.persist_and_notify_client_events(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/handlers/message.py", line 1912, in persist_and_notify_client_events
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ) = await self._storage_controllers.persistence.persist_events(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 421, in persist_events
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/util/async_helpers.py", line 307, in yieldable_gather_results
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: raise dfe.subFailure.value from None
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: result = context.run(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: return g.throw(self.type, self.value, self.tb)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 416, in enqueue
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: return await self._event_persist_queue.add_to_queue(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 239, in add_to_queue
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: res = await make_deferred_yieldable(end_item.deferred.observe())
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 282, in handle_queue_loop
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ret = await self._per_item_callback(room_id, item.task)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 362, in _process_event_persist_queue_task
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: return await self._persist_event_batch(room_id, task)
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 610, in _persist_event_batch
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ) = await self._calculate_new_forward_extremities_and_state_delta(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 690, in _calculate_new_forward_extremities_and_state_delta
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: res = await self._get_new_state_after_events(
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 876, in _get_new_state_after_events
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: old_state_groups = {
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: File "/usr/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 877, in
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: event_id_to_state_group[evid] for evid in old_latest_event_ids
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: KeyError: '$sX7ySz3cOPShqZYA2hMRulsUIw3-Rtgdtxv5isvUC9o'
Dec 18 20:23:07 serverOS-kimiblock.top synapse[566873]: 2023-12-18 20:23:07,061 - synapse.access.http.60000 - 465 - INFO - POST-2638226 - 127.0.0.1 - 60000 - {@kimiblockmoe:moechat.kimiblock.top} Processed request: 0.043sec/0.001sec (0.007sec, 0.003sec) (0.003sec/0.012sec/9) 55B 500 "POST /_matrix/client/v3/rooms/!ReTqqPGlVRRlnqJPJx%3Aarchlinux.org/leave HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0" [0 dbevts]
```

### Anything else that would be useful to know?

The latest room state in Synapse is stuck at 7 Dec 2023, 9:52:14 pm, with the source of:

```
{
"content": {
"body": "> <@fr3ax:matrix.org> oh I can now make my own pixrobots as long as I don't sell them and contribute back w my improvements?\n\nu can make ur own pixrobot's and even sell them as long as you make the source _**code**_ available to the _user_",
"format": "org.matrix.custom.html",
"formatted_body": "

In reply to @fr3ax:matrix.org
oh I can now make my own pixrobots as long as I don't sell them and contribute back w my improvements?

u can make ur own pixrobot's and even sell them as long as you make the source code available to the user",
"m.mentions": {
"user_ids": [
"@fr3ax:matrix.org"
]
},
"m.relates_to": {
"m.in_reply_to": {
"event_id": "$lCm8ehVZeQWiId34ydIevecBev5vxQkJoAwem1Xkhlk"
}
},
"msgtype": "m.text"
},
"origin_server_ts": 1701957134886,
"sender": "@morguldir:sulian.eu",
"type": "m.room.message",
"unsigned": {},
"event_id": "$V5G7ry-WY6sdUgvK9aKqQ4qgb0TWjZK7CrQNZ4uZDcs",
"room_id": "!ReTqqPGlVRRlnqJPJx:archlinux.org"
}
```

Sadly I don't have logs on 7 Dec.