element-hq / element-hq/synapse
Crashes on startup while generating presence stream
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#10106](https://github.com/matrix-org/synapse/issues/10106).
---
### Description
Crashes on startup while generating presence stream
### Steps to reproduce
- HS is running a set of workers: event-persister, federation-{inbound,reader,sender}, presence-writer
- All workers and main process were restarted
- event-persister and federation-sender crashed 3 times each on startup before successfully starting
Event Persister:
```
2021-06-02 07:46:12,318 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,318 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,319 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for public_room_list_stream(stream_id)
2021-06-02 07:46:12,321 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for push_rules_stream(stream_id)
2021-06-02 07:46:12,327 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,327 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,328 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for local_group_updates(stream_id)
2021-06-02 07:46:12,330 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_stream(stream_id)
2021-06-02 07:46:12,331 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for user_signature_stream(stream_id)
2021-06-02 07:46:12,331 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_outbound_pokes(stream_id)
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- Traceback (most recent call last):
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- return _run_code(code, main_globals, None,
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel- exec(code, run_globals)
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 483, in
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- start(sys.argv[1:])
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 470, in start
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- hs.setup()
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/server.py", line 280, in setup
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- self.datastores = Databases(self.DATASTORE_CLASS, self)
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 76, in __init__
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- main = main_store_class(database, db_conn, hs)
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 553, in __init__
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 40, in __init__
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/stats.py", line 103, in __init__
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/presence.py", line 46, in __init__
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- self._presence_id_gen = MultiWriterIdGenerator(
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 283, in __init__
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel- self._load_current_ids(db_conn, tables)
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 304, in _load_current_ids
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel- cur.execute(sql, (self._stream_name, self._writers))
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 290, in execute
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- self._do_execute(self.txn.execute, sql, *args)
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 316, in _do_execute
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- return func(sql, *args)
2021-06-02 07:46:12,351 - twisted - 258 - ERROR - sentinel- psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
```
Federation Sender:
```
2021-06-02 07:46:12,819 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,820 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,821 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for public_room_list_stream(stream_id)
2021-06-02 07:46:12,821 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for push_rules_stream(stream_id)
2021-06-02 07:46:12,826 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,826 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,827 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for local_group_updates(stream_id)
2021-06-02 07:46:12,829 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_stream(stream_id)
2021-06-02 07:46:12,829 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for user_signature_stream(stream_id)
2021-06-02 07:46:12,830 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_outbound_pokes(stream_id)
2021-06-02 07:46:12,841 - twisted - 258 - ERROR - sentinel- Traceback (most recent call last):
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel- return _run_code(code, main_globals, None,
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- exec(code, run_globals)
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/federation_sender.py", line 23, in
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- start(sys.argv[1:])
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 470, in start
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel- hs.setup()
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/server.py", line 280, in setup
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel- self.datastores = Databases(self.DATASTORE_CLASS, self)
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 76, in __init__
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel- main = main_store_class(database, db_conn, hs)
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 553, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 40, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/stats.py", line 103, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/presence.py", line 46, in __init__
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- self._presence_id_gen = MultiWriterIdGenerator(
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 283, in __init__
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- self._load_current_ids(db_conn, tables)
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 304, in _load_current_ids
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- cur.execute(sql, (self._stream_name, self._writers))
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 290, in execute
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- self._do_execute(self.txn.execute, sql, *args)
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 316, in _do_execute
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- return func(sql, *args)
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
```
Raising as this HS is testing out the presence worker and the presence stream appears to be the problematic one.
### Version information
- **Homeserver**: vector.modular.im
- **Version**: 1.34.0
Contributor guide
Assessment
This issue has not been assessed yet.