element-hq / element-hq/synapse

Crashes on startup while generating presence stream

Open
#10,106 0 comments 0 reactions 0 assignees View on GitHub
S-Minor T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#10106](https://github.com/matrix-org/synapse/issues/10106).

---

### Description

Crashes on startup while generating presence stream

### Steps to reproduce

- HS is running a set of workers: event-persister, federation-{inbound,reader,sender}, presence-writer
- All workers and main process were restarted
- event-persister and federation-sender crashed 3 times each on startup before successfully starting

Event Persister:
```
2021-06-02 07:46:12,318 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,318 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,319 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for public_room_list_stream(stream_id)
2021-06-02 07:46:12,321 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for push_rules_stream(stream_id)
2021-06-02 07:46:12,327 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,327 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,328 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for local_group_updates(stream_id)
2021-06-02 07:46:12,330 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_stream(stream_id)
2021-06-02 07:46:12,331 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for user_signature_stream(stream_id)
2021-06-02 07:46:12,331 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_outbound_pokes(stream_id)
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- Traceback (most recent call last):
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- return _run_code(code, main_globals, None,
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel- exec(code, run_globals)
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 483, in
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- start(sys.argv[1:])
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 470, in start
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- hs.setup()
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/server.py", line 280, in setup
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- self.datastores = Databases(self.DATASTORE_CLASS, self)
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 76, in __init__
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- main = main_store_class(database, db_conn, hs)
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 553, in __init__
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 40, in __init__
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/stats.py", line 103, in __init__
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/presence.py", line 46, in __init__
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- self._presence_id_gen = MultiWriterIdGenerator(
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 283, in __init__
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel- self._load_current_ids(db_conn, tables)
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 304, in _load_current_ids
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel- cur.execute(sql, (self._stream_name, self._writers))
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 290, in execute
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- self._do_execute(self.txn.execute, sql, *args)
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 316, in _do_execute
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel- return func(sql, *args)
2021-06-02 07:46:12,351 - twisted - 258 - ERROR - sentinel- psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
```

Federation Sender:
```
2021-06-02 07:46:12,819 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,820 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,821 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for public_room_list_stream(stream_id)
2021-06-02 07:46:12,821 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for push_rules_stream(stream_id)
2021-06-02 07:46:12,826 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,826 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,827 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for local_group_updates(stream_id)
2021-06-02 07:46:12,829 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_stream(stream_id)
2021-06-02 07:46:12,829 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for user_signature_stream(stream_id)
2021-06-02 07:46:12,830 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_outbound_pokes(stream_id)
2021-06-02 07:46:12,841 - twisted - 258 - ERROR - sentinel- Traceback (most recent call last):
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel- return _run_code(code, main_globals, None,
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- exec(code, run_globals)
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/federation_sender.py", line 23, in
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- start(sys.argv[1:])
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 470, in start
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel- hs.setup()
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/server.py", line 280, in setup
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel- self.datastores = Databases(self.DATASTORE_CLASS, self)
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 76, in __init__
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel- main = main_store_class(database, db_conn, hs)
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 553, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 40, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/stats.py", line 103, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel- super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/presence.py", line 46, in __init__
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- self._presence_id_gen = MultiWriterIdGenerator(
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 283, in __init__
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel- self._load_current_ids(db_conn, tables)
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 304, in _load_current_ids
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- cur.execute(sql, (self._stream_name, self._writers))
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 290, in execute
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel- self._do_execute(self.txn.execute, sql, *args)
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 316, in _do_execute
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- return func(sql, *args)
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update
```

Raising as this HS is testing out the presence worker and the presence stream appears to be the problematic one.

### Version information

- **Homeserver**: vector.modular.im
- **Version**: 1.34.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.