element-hq / element-hq/synapse
Rooms created without any membership (only `m.room.create` event)
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#15005](https://github.com/matrix-org/synapse/issues/15005).
---
### Description
Trying to tie up any loose-ends with the Gitter import as part of the [Gitter migration process](https://github.com/vector-im/roadmap/issues/26) and noticing that we have some Matrix rooms recorded that seem to broken on the Synapse side. Some of these rooms only have a `m.room.create` event in it while others have no events which means we can't interact with it at all since permission denied errors everywhere (no membership or power-level events).
~~The only way we recorded this room on the Gitter side is if we got a 200-response from Synapse originally when it was created. If Synapse is returning with a error-free response, there should be much better guarantees on the room being created successfully (atomic transaction). Of course it's possible this could have been corrupted after the fact though.~~ What most likely happened was the rooms were attempted to be created but timed out. Then we introduced code to reconcile the case where a Matrix room alias exists but we don't have the bridged Matrix room entry recorded, see https://github.com/matrix-org/synapse/issues/15005#issuecomment-1421481395
They all seem to be created on January 10th, 2023 which is probably in the middle of importing messages and the import scripts.
Here is the list of affected rooms that I know of:
- `!gHbYoGxiEuXmahoYeh:gitter.im` (only `m.room.create`)
- `!OpjflmXuauidodzmJH:gitter.im` (only `m.room.create`)
- `!QCVddUkjrHLjEqeXQd:gitter.im` (no events)
- `!wJBkwhqwLssHVjKErI:gitter.im` (only `m.room.create`)
- `!kkjacXiigblOffltrw:gitter.im` (only `m.room.create`)
- `!etJKlYVRXEtkkMdEOJ:gitter.im` (no events)
- `!VeVZEnGasXJXpAYECh:gitter.im` (only `m.room.create`)
- `!iJIbqcgSzsAFnwjVBI:gitter.im` (no events)
- `!koFtykttKGRpjbzflq:gitter.im` (no events)
- `!mxfIojpZcyQDBbmmoY:gitter.im` (only `m.room.create`)
- `!ezxePTXYHMSIkpGOiy:gitter.im` (only `m.room.create`)
- `!QlEqqNHuTSIHswllOX:gitter.im` (only `m.room.create`)
- `!XInKHVcDAmmyGBEUwr:gitter.im` (no events)
- `!KIKPFcoDnhBeZhLNUr:gitter.im` (only `m.room.create`)
- `!BcBAOZpxjTAvMDfSHf:gitter.im` (only `m.room.create`)
- `!DuyTBdNdlhfXoBFNqr:gitter.im` (only `m.room.create`)
- `!qgfkmcpByUudrFHTMX:gitter.im` (no events)
- `!tFwVYIoUNZQhRhjFNO:gitter.im` (no events)
- `!qoZZIEjvblGKIkqEdj:gitter.im` (no events)
---
In terms of moving Gitter forward, I think we can safely reconcile this by removing the bridged room entry in the Gitter database and the Matrix room alias and have them re-created (tracked by https://gitlab.com/gitterHQ/webapp/-/issues/2858#note_1267698515). But I'm unable to remove the Matrix room alias from the old room (`M_FORBIDDEN`) 😬
And it would still be good to fix the Synapse bugs that allow this to happen.
### Steps to reproduce
1. Not sure how this happened but it's all Gitter appservice interactions against Synapse
### Homeserver
gitter.im
### Synapse Version
1.74.0
### Installation Method
I don't know
### Database
PostgreSQL
### Workers
Multiple workers
### Platform
`gitter.im` (`gitter.ems.host`) runs on EMS but I don't know the details
Python version: `3.11.1`
### Configuration
_No response_
### Relevant log output
The only relevant logs for this room from the time of creation is this message:
https://modular-euwest2-kibana.proxy.matrix.org/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-8w,to:now-3w))&_a=(columns:!(_source),index:'67745970-aa2d-11ea-96a7-efc84863d0ea',interval:auto,query:(language:kuery,query:'kubernetes.pod.labels.hostname:%22gitter.ems.host%22%20AND%20message:%22!!gHbYoGxiEuXmahoYeh:gitter.im%22'),sort:!(!('@timestamp',desc)))
```
2023-01-09 17:33:53,203 - synapse.replication.http.send_events - 149 - INFO - POST-4249- Got batch of events to send, last ID of batch is: $JeDmcg6bYb0s-d6pxIbgmfPaJoChH-qWlnTZ3n8xGdU, sending into room: !gHbYoGxiEuXmahoYeh:gitter.im
```
---
Database information:
```
# Connect to the database: https://gitlab.matrix.org/new-vector/internal/-/wikis/EMS-postgres#connecting-to-an-ems-database
$ ems-db-debug gitter.im
select * FROM rooms WHERE room_id = '!gHbYoGxiEuXmahoYeh:gitter.im';
room_id | is_public | creator | room_version | has_auth_chain_index
-------------------------------+-----------+--------------------------+--------------+----------------------
!gHbYoGxiEuXmahoYeh:gitter.im | t | @gitter-badger:gitter.im | 9 | t
select * FROM room_memberships WHERE room_id = '!gHbYoGxiEuXmahoYeh:gitter.im';
event_id | user_id | sender | room_id | membership | forgotten | display_name | avatar_url
----------+---------+--------+---------+------------+-----------+--------------+------------
(0 rows)
select * FROM events WHERE room_id = '!gHbYoGxiEuXmahoYeh:gitter.im';
topological_ordering | event_id | type | room_id | content | unrecognized_keys | processed | outlier | depth | origin_server_ts | received_ts |
sender | contains_url | instance_name | stream_ordering | state_key | rejection_reason
----------------------+----------------------------------------------+---------------+-------------------------------+---------+-------------------+-----------+---------+-------+------------------+---------------+-------
-------------------+--------------+---------------------------+-----------------+-----------+------------------
1 | $JeDmcg6bYb0s-d6pxIbgmfPaJoChH-qWlnTZ3n8xGdU | m.room.create | !gHbYoGxiEuXmahoYeh:gitter.im | | | t | f | 1 | 1673285632772 | 1673285633224 | @gitte
r-badger:gitter.im | f | synapse-event-persister-3 | 94262741 | |
(1 row)
select * FROM room_aliases WHERE room_id = '!gHbYoGxiEuXmahoYeh:gitter.im';
room_alias | room_id | creator
----------------------------------+-------------------------------+--------------------------
#GreenShoeGarage_Lobby:gitter.im | !gHbYoGxiEuXmahoYeh:gitter.im | @gitter-badger:gitter.im
(1 row)
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.