element-hq / element-hq/synapse

`KeyError: ('m.room.power_levels', '')` when joining some rooms

Open
#19,679 1 comment 0 reactions 0 assignees View on GitHub
A-Federated-Join A-Join
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

Some room joins seem to fail with `KeyError: ('m.room.power_levels', '')` almost immediately after `send_join` returns, presenting a 500 Internal Server Error to the user, and ultimately preventing the join.
Sometimes retrying the join a few more times succeeds, but increasingly commonly (like in the room in the log snippet) it does not.

### Steps to reproduce

- join a remote room for the first time

The error is too vague for me to be able to provide any meaningful reproduction steps. All I did in this case was attempted to join `#element-x-ios:matrix.org` as I would any other room.

### Homeserver

starstruck.systems

### Synapse Version

1.151.0rc1

### Installation Method

pip (from PyPI)

### Database

postgres, mono, clean

### Workers

Multiple workers

### Platform

Ubuntu LXC, master process and worker processes controlled by systemd, same-metal postgres in another lxc, redis on the same lxc hosting synapse

### Configuration

Many experimental features enabled, presence disabled, synapse-http-antispam module enabled - full config at https://starstruck.systems/~star/synapse.yaml

### Relevant log output

```shell
2026-04-10 18:52:00,459 INFO synapse.http.matrixfederationclient (POST-149674): {GET-O-18866} [matrix.org] Completed request: 200 OK in 0.11 secs, got 3088 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/query/directory?room_alias=%23element-x-ios%3Amatrix.org
2026-04-10 18:52:00,620 INFO synapse.http.matrixfederationclient (POST-149674): {GET-O-18867} [matrix.org] Completed request: 200 OK in 0.16 secs, got 482 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/make_join/%21nqXEcZbBVWpiHLmQhVANyKwuRD3d672nXjd1iEKPuTU/%40nex%3Astarstruck.systems?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11&ver=12&ver=org.matrix.msc3757.10&ver=org.matrix.msc3757.11&ver=org.matrix.hydra.11
2026-04-10 18:52:01,413 INFO synapse.http.matrixfederationclient (POST-149674): {PUT-O-18868} [matrix.org] Completed request: 200 OK in 0.79 secs, got 1382050 bytes - PUT matrix-federation://matrix.org/_matrix/federation/v2/send_join/%21nqXEcZbBVWpiHLmQhVANyKwuRD3d672nXjd1iEKPuTU/%24z5m1ksllJoTTzPw73rFDX3JMO9hKy1vsPRjP-Xa3Ezc
2026-04-10 18:52:01,413 INFO synapse.federation.federation_client (POST-149674): Processing from send_join 1860 events
2026-04-10 18:52:01,830 INFO synapse.crypto.keyring (PerspectivesKeyFetcher-72): Requesting keys <...>
2026-04-10 18:51:30,223 INFO synapse.handlers.federation_event (POST-149621): Persisting 753 remaining outliers: ['$iCgYS2KvQ03DJQtygE5uUdbeGvMPV954nO_zhL1nR6Q', '$iCt7sazLa9DSjy341dhUkX9JlV8p6M6-XOfTNdAzn3U', '$iDw578dOuZsG1qmGnOKo7CG6N1bD85Tz1IPQPmZ9a50', '$86fslg5C3fnhbVdv26Rw_JcivdfrpJUTwDQzLBkrrCQ', '$iJJQee_0dFewEva3dSWuMTvwpd4FtUZaBWysL64s7Po', ...]
2026-04-10 18:51:30,516 ERROR synapse.http.server (POST-149621): Failed handle request via 'JoinRoomAliasServlet':
Traceback (most recent call last):
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/http/server.py", line 335, in _async_render_wrapper
callback_return = await self._async_render(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/http/server.py", line 576, in _async_render
callback_return = await raw_callback_return
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/rest/client/room.py", line 615, in on_POST
return await self._do(request, requester, room_identifier, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/rest/client/room.py", line 596, in _do
await self.room_member_handler.update_membership(
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/room_member.py", line 670, in update_membership
result = await self.update_membership_locked(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/room_member.py", line 1111, in update_membership_locked
remote_join_response = await self._remote_join(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/room_member.py", line 1970, in _remote_join
event_id, stream_id = await self.federation_handler.do_invite_join(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation.py", line 716, in do_invite_join
await self._federation_event_handler.process_remote_join(
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation_event.py", line 570, in process_remote_join
await self._auth_and_persist_outliers(
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation_event.py", line 1825, in _auth_and_persist_outliers
await self.persist_events_and_notify(
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation_event.py", line 2331, in persist_events_and_notify
) = await self._storage_controllers.persistence.persist_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 435, in persist_events
ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/util/async_helpers.py", line 347, in yieldable_gather_results
raise dfe.subFailure.value from None
File "/opt/synapse/.venv/lib/python3.12/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
return g.throw(self.value.with_traceback(self.tb))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 430, in enqueue
return await self._event_persist_queue.add_to_queue(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 251, in add_to_queue
res = await make_deferred_yieldable(end_item.deferred.observe())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/twisted/internet/defer.py", line 1187, in __iter__
yield self
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 294, in handle_queue_loop
ret = await self._per_item_callback(room_id, item.task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 376, in _process_event_persist_queue_task
return await self._persist_event_batch(room_id, task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 661, in _persist_event_batch
await self.persist_events_store._persist_events_and_state_updates(
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/databases/main/events.py", line 462, in _persist_events_and_state_updates
redact_allowed = await self._can_sender_redact(event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/databases/main/events.py", line 502, in _can_sender_redact
pl_id = state[(EventTypes.PowerLevels, "")]
~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/databases/main/state.py", line 984, in __getitem__
return super().__getitem__(key)
^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: ('m.room.power_levels', '')
2026-04-10 18:51:30,518 INFO synapse.access.http.http (POST-149621): 80.3.155.46 - http - {@nex:starstruck.systems} Processed request: 14.069sec/0.000sec ru=(0.318sec, 0.007sec) db=(0.002sec/0.019sec/15) 67B 500 "POST /_matrix/client/v3/join/%23element-x-ios:matrix.org HTTP/1.1" "gomuks/v26.03+dev.ce0c1d61 mautrix-go/v0.26.4+dev.261f1770 go/1.26.1" [1 dbevts]
```

### Anything else that would be useful to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.